How to Choose an LLM for Your Product

There is no best LLM for your product. There is a best model for each task your product performs, measured on your own data, at a cost and latency your users and margins can live with, from a provider whose data terms and platform fit your customers. And that answer will change within months, because model releases, price cuts, and retirements now arrive at a pace no annual vendor review can match.

So the goal is not to pick a winner once. It is to build a selection process you can rerun cheaply whenever the landscape shifts. This guide covers the criteria that actually decide the choice, a step-by-step process, how model tiers map to tasks, when open-weight models make sense, and how to design for switching from day one.

It is part of our series on adding AI features to an existing product.

Why “which model is best” is the wrong question

Different tasks need different models. A single feature might classify an incoming request, retrieve context, draft an answer, and check it. Classification rarely needs a frontier model; a nuanced multi-step answer might. Picking one model for everything means overpaying for the easy work or underdelivering on the hard work.

Public leaderboards are a weak signal for your use case. They measure general preferences or generic benchmarks, not your documents, your users, or your definition of a good answer. They can also be distorted. Researchers auditing Chatbot Arena, one of the most cited leaderboards, found that some providers privately tested many model variants before release and could withhold unfavorable results; in one case, a provider tested 27 private variants ahead of a launch [1]. The authors also found proprietary models were sampled more often than open alternatives, giving them more data to tune against [1]. Leaderboards are useful for building a shortlist. They should not make the decision.

The same open-weight model can behave differently depending on who hosts it. When independent testers ran OpenAI’s gpt-oss-120b across hosting providers, scores on the same benchmark varied noticeably; one major cloud’s lower result traced back to an outdated serving configuration that ignored the reasoning setting [2]. If you evaluate an open-weight model, evaluate it on the exact provider and configuration you will run in production.

The six criteria that decide the choice

Three of these deserve more detail.

Cost per successful task, not per token. Token counts for the same text differ by tokenizer. Anthropic notes that its newer Claude models produce roughly 30% more tokens for identical text than earlier ones [4]. Reasoning models also generate hidden reasoning tokens that are billed as output. A model with a lower list price can cost more per completed task, and a pricier model that passes on the first attempt can be cheaper than a cheap one users regenerate. Our guide to LLM API costs in production shows how to model this.

Data terms depend on the tier, not just the provider. Google’s Gemini API pricing page is a clear example: content sent on the free tier may be used to improve Google’s products, while content on the paid tier is not [6]. Prototype on free tiers with synthetic data only, and confirm the terms of the exact tier and platform you will ship on. Where the model runs matters too; see Azure OpenAI vs OpenAI API vs AWS Bedrock.

Lifecycle is a real risk. Models get retired, and requests to a retired model fail rather than falling back. Anthropic, for example, commits to at least 60 days’ notice before retiring publicly released models on its own platforms, publishes tentative “not sooner than” retirement dates for current models, and notes that partner platforms such as Amazon Bedrock and Google Cloud set their own retirement schedules [7]. Whatever provider you choose, find its deprecation policy before you ship, and assume you will migrate at least once a year.

A seven-step selection process

1. List tasks, not features. Break each AI feature into the distinct model calls it needs: classify, extract, retrieve and answer, draft, check, act.
2. Define pass criteria per task. Decide what a correct output looks like and how costly a wrong one is. A drafted reply the user edits tolerates more error than an extracted invoice amount.
3. Build an eval set from real data. Use 50 to 200 anonymized real inputs per task, including known edge cases. As Hamel Husain argues, products without a robust evaluation system tend to fail regardless of model choice [8].
4. Shortlist across tiers and at least two providers. Include a frontier, a mid-tier, and a small model, from more than one vendor, so you learn where the quality curve flattens for your task and have a credible fallback.
5. Run the evals and record quality, cost, and latency together. Use identical prompts adapted only where a model needs it, and run enough samples to see variance.
6. Pick the smallest model that clears the bar for each task, and name a fallback. The fallback should come from a different provider and pass the same evals, even if it costs a bit more.
7. Set re-evaluation triggers. Rerun the suite when a relevant model launches, when a provider changes prices, when a deprecation notice arrives, and at least quarterly.

Here is what a scorecard for a single task might look like. The models and numbers are hypothetical, for illustration only:

In this example, the frontier model buys one point of accuracy at more than four times the cost, so the mid-tier model wins, and the small model becomes a candidate again only if prompt or retrieval changes lift it over the bar.

Match the model tier to the task

Providers now ship model families in explicit tiers. The names change constantly; the pattern does not. List prices below are per million input and output tokens as of September 11, 2026.

Two practical notes. First, prices are not comparable across rows at face value because tokenizers, reasoning behavior, and output length differ; only your eval run tells you cost per task. Second, most products end up using two or three tiers at once, with a router sending each request to the cheapest model that handles it. We cover routing and caching in LLM API costs in production.

Closed vs open-weight models

Open-weight models, whose weights you can download and run yourself or through a hosting provider, have become genuinely competitive. Epoch AI found that since January 2026, the most capable open-weight models have trailed the best closed models by an average of about four months on its aggregate capability index [3]. Families from DeepSeek, Qwen, Moonshot’s Kimi, and Zhipu’s GLM are available on major clouds as fully managed endpoints [10].

Open weights make sense when:

• You need full control over where data is processed, including on-premises or air-gapped deployments.
• You run very high, steady volume where dedicated infrastructure beats per-token pricing.
• You need deep customization through fine-tuning that closed providers do not offer for your use case.
• You want insurance against a single vendor’s pricing or retirement decisions.

The tradeoffs are real:

• Running models yourself means owning GPU capacity, serving software, scaling, and upgrades.
• Quality depends on the serving stack, as the hosted gpt-oss results showed [2].
• You take on more responsibility for safety behavior, content controls, and license review.
• A four-month average gap still matters for tasks at the edge of current capability.

For most teams adding their first AI features, a closed model through an API or cloud platform is the faster path to production, with an open-weight model evaluated as a fallback or for specific high-volume tasks.

Plan for change from day one

The one prediction you can make with confidence is that your current choice will not last. Prices alone moved in both directions within weeks this summer:

• OpenAI cut GPT-5.6 Luna’s price by 80% and Terra’s by 20% on July 30, 2026, roughly three weeks after launch [5], and replaced its Priority Processing option with a new Fast mode priced at twice the standard rate [9].
• Google’s Gemini 3.8 Flash launched at introductory pricing that doubles on January 1, 2027 [6].
• Anthropic made Claude Sonnet 5’s introductory price permanent and cancelled a scheduled increase [4].

Add new model releases and retirements, and the practical conclusion is clear: switching models should be a configuration change plus an eval run, not a rewrite. That requires three things you should build before your first launch:

• A model gateway between product code and providers, with prompts and model IDs in configuration.
• An eval suite per task that runs in CI and against any candidate model on demand.
• Telemetry that records model, tokens, latency, cost, and outcome for every call, so you can compare a new model against real production behavior.

With those in place, a price cut becomes a same-week saving and a deprecation notice becomes a routine ticket. We cover fallbacks between providers in making LLM features reliable.

Common mistakes

• Choosing a model because it topped a leaderboard or won a demo.
• Using one frontier model for every task in a feature.
• Comparing per-token prices instead of cost per successful task.
• Prototyping on a free tier with real customer data.
• Evaluating an open-weight model on one host and deploying on another.
• Shipping without a named fallback from a second provider.
• Treating model selection as a one-time decision.

Model selection checklist

• Each AI feature is broken into discrete tasks with written pass criteria.
• Every task has an eval set built from real, anonymized product data.
• Candidates span at least three tiers and two providers.
• Quality, cost per successful task, and latency were measured together.
• The data terms of the exact tier, platform, and region are confirmed.
• The provider’s deprecation policy and your model’s retirement status are documented.
• A fallback model from a different provider passes the same evals.
• Model IDs live in configuration behind a gateway, with re-evaluation triggers defined.

Keep reading

• How to add AI features to an existing software product (the pillar for this series)
• LLM API costs in production: how to estimate and cut them
• Azure OpenAI vs OpenAI API vs AWS Bedrock for enterprise apps
• Latency, rate limits, and fallbacks: making LLM features reliable
• Our enterprise RAG guide

CMW AI Labs helps software teams run model selection the way this guide describes: task-level evals on real data, multi-provider shortlists, and gateways that make the next switch cheap. If you are choosing models for a new AI feature, talk to our team.

Sources

1. Singh et al., “The Leaderboard Illusion,” arXiv:2504.20879 (NeurIPS 2025). https://arxiv.org/abs/2504.20879
2. Simon Willison, “Open weight LLMs exhibit inconsistent performance across providers,” August 15, 2025. https://simonwillison.net/2025/Aug/15/inconsistent-performance/
3. Epoch AI, “Open models lag state-of-the-art closed models by 4 months,” May 29, 2026. https://epoch.ai/data-insights/open-closed-eci-gap
4. Anthropic, “Pricing,” Claude Platform Docs (accessed September 2026). https://platform.claude.com/docs/en/about-claude/pricing
5. Quartz, “OpenAI cuts prices for GPT-5.6 Luna and Terra,” July 30, 2026. https://qz.com/openai-price-cuts-gpt-luna-terra-073026
6. Google, “Gemini Developer API pricing” (accessed September 2026). https://ai.google.dev/gemini-api/docs/pricing
7. Anthropic, “Model deprecations,” Claude Platform Docs. https://platform.claude.com/docs/en/resources/model-deprecations
8. Hamel Husain, “Your AI Product Needs Evals.” https://hamel.dev/blog/posts/evals/
9. Let’s Data Science, “OpenAI Cuts GPT-5.6 Luna and Terra Prices,” July 2026. https://letsdatascience.com/news/openai-cuts-gpt-56-luna-and-terra-prices-ae12009d
10. AWS What’s New, “Amazon Bedrock adds support for six fully-managed open weights models,” February 10, 2026. https://aws.amazon.com/about-aws/whats-new/2026/02/amazon-bedrock-adds-support-six-open-weights-models