LLM API Costs in Production: How to Estimate and Cut Them

The cost of an LLM feature is input tokens times the input price, plus cached tokens times the cached price, plus output tokens (including hidden reasoning tokens) times the output price, multiplied by the number of model calls behind every user action. The formula is simple. The trouble is that first estimates usually count one call with a short prompt, while production runs several calls per action with long, growing context, retries, and regenerations.

Getting this right matters more in 2026 than it did during the pilot era. Gartner reported in July 2026 that enterprise AI budgets are under growing scrutiny, with spending shifting toward vendors that can demonstrate value on cost, latency, performance, and reliability [1]. If you are adding AI to an existing product, your unit economics are now part of the sales conversation.

This guide explains how pricing works across the major providers, where the hidden multipliers come from, how to build a cost model before you ship, and which levers cut spend without degrading quality. It is part of our series on adding AI features to an existing product.

How LLM API pricing actually works

Every major provider bills by the token, but several categories of token carry different prices, and a handful of modifiers sit on top.

Tokens. A token is a chunk of text, roughly four characters or three quarters of a word in English [2]. Token counts are not comparable across models, because each model family uses its own tokenizer. Anthropic notes, for example, that its Claude 4.7 and later models use a newer tokenizer that produces about 30% more tokens for the same text than earlier models [2]. The practical consequence: compare models on cost per completed task, measured on your own data, not on the per-token price column.

Input versus output. Output tokens cost several times more than input tokens. Across Anthropic’s current lineup, output is priced at five times input; Claude Sonnet 5, for instance, lists at $2 per million input tokens and $10 per million output tokens [2]. Verbose responses are expensive responses.

Reasoning tokens. Reasoning models think before answering, and that thinking is billed. OpenAI states that reasoning tokens are not shown in the answer but are counted and billed as output tokens [3]. Microsoft’s Azure documentation adds that a single request can consume anywhere from a few hundred to tens of thousands of reasoning tokens depending on problem difficulty [4]. A two-sentence answer can carry a large invisible bill.

Cached input. When the start of a prompt repeats across requests, providers bill the repeated portion at a steep discount. OpenAI applies caching automatically to prompts of 1,024 tokens or more, matching on exact prefixes, and says it can cut input token costs by up to 90% [5]. Anthropic prices cache reads at 0.1x the base input rate and cache writes at 1.25x for a 5-minute cache or 2x for a 1-hour cache, which means a 5-minute cache pays for itself after a single read [2].

Batch processing. Work that does not need an immediate answer can go through batch APIs at half price. OpenAI, Anthropic, and Google all offer a 50% batch discount [6][2][7], and Anthropic notes that batch and caching discounts stack [2].

Modifiers. Tool definitions and schemas count as input tokens on every call that includes them, and some server-side tools carry their own fees; web search on the Claude API, for example, costs $10 per 1,000 searches on top of tokens [2]. Data residency can add a premium as well: Anthropic applies a 1.1x multiplier for US-only inference on recent models, and Claude’s regional endpoints on Bedrock and Google Cloud carry a 10% premium over global endpoints [2].

For how these prices differ once you route through a cloud platform, see Azure OpenAI vs OpenAI API vs AWS Bedrock.

Anthropic cache reads, against base input price

0.1x

A write costs 1.25x, so a 5-minute cache pays for itself after one read.

OpenAI's stated ceiling on input cost reduction from caching

90%

Applied automatically to prompts of 1,024 tokens or more.

Batch API discount

50%

Offered by OpenAI, Anthropic and Google. Stacks with caching.

Why production bills surprise teams

The per-token math is rarely where estimates go wrong. The multipliers are.

Multiple calls per user action. A single “answer this question” feature often runs a query rewrite, retrieval, the main generation, and a guardrail or grading check. That is three or four model calls billed for one click.

Conversation history. Chat-style features resend the full conversation on every turn, so the input for turn ten includes turns one through nine. Cost per turn climbs as sessions get longer.

Agent loops. Agents call the model repeatedly, and each step typically resends the growing context plus tool results. A task that takes fifteen steps can cost fifteen times a single call, or more. We cover when agents are worth that in AI agents in the enterprise.

Retrieved context. Retrieval-augmented features routinely stuff thousands of tokens of documents into each prompt, much of it irrelevant. Better retrieval precision is a cost lever, not just a quality one. See our enterprise RAG guide.

Retries and regenerations. Timeouts, malformed structured outputs, and users clicking “regenerate” all produce billed calls that never show up in a happy-path estimate.

Reasoning defaults. Using a reasoning model, or a high reasoning setting, for tasks that do not need it quietly multiplies output spend.

Tokenizer changes. Switching models can change token counts for identical text, so a lower per-token price does not guarantee a lower bill [2].

Non-product usage. Eval runs, LLM-as-judge scoring, staging traffic, and re-embedding a corpus when you change embedding models all consume budget that no customer sees.

Meanwhile, the price of a fixed level of capability keeps falling. Epoch AI estimates that the cost to reach a given benchmark performance has dropped by roughly 9x to 900x per year depending on the threshold [8]. That is real, but it does not mean your bill falls. Teams tend to spend the savings on bigger models, longer context, more reasoning, and more features, so cost discipline has to be designed in rather than waited for.

How to estimate cost before you build

A cost model you can defend takes an afternoon, not a spreadsheet project.

1. Map the call graph. For each user action, list every model call it triggers, including rewrites, guardrails, and graders.
2. Measure real token counts. Run 50 to 100 real inputs through a prototype and record input, cached, output, and reasoning tokens from the API’s usage fields. Anthropic offers a token counting endpoint that estimates a request’s tokens before you send it [2]; for OpenAI models, the tiktoken library does the same job locally.
3. Calculate cost per action. Apply current list prices to each token category and sum across calls.
4. Apply behavior multipliers. Include regeneration rate, retry rate, and average conversation length.
5. Model volume by tenant, not just in total. Usage in B2B products is rarely even. A few heavy accounts can generate a large share of calls, and that distribution decides whether your pricing works.
6. Revisit after beta. Replace every assumption with measured production numbers once real users arrive.

A worked example: AI reply drafting in a helpdesk product

Assume a B2B helpdesk product adds a “draft reply” button. The feature runs 500,000 drafts per month across all customers. Prices below are Anthropic’s published list prices as of September 2026 [2]; the method works identically for any provider.

Each request contains a 3,000-token static prefix (instructions, tone guidelines, tool definitions), 2,500 tokens of ticket thread and customer record, 3,000 tokens of retrieved help-center articles, and produces a 350-token draft. Roughly 20% of drafts get regenerated once.

Baseline, everything on Claude Sonnet 5 with no optimization. Input is 8,500 tokens at $2 per million, and output is 350 tokens at $10 per million, for about $0.0205 per draft. With regenerations, that comes to roughly $12,300 per month.

Now apply the levers one at a time:

That is roughly a 59% reduction without touching the feature’s scope. Three assumptions deserve scrutiny before you rely on numbers like these. First, cache savings assume steady traffic keeps the cache warm; if you put tenant-specific content in the prefix, each tenant gets its own cache, and low-volume tenants will pay write premiums more often. Second, the routing step is only valid if your evals show the smaller model meets the quality bar on the tickets it receives. The figure also includes the classifier’s own call, which is why it is not simply a blend of the two model prices. Third, token counts were held constant across models for simplicity, even though tokenizers differ.

The per-tenant view matters just as much. At optimized cost, an average customer sending 500 drafts a month costs about $5, while a heavy customer sending 20,000 drafts costs about $200. At baseline, that heavy customer would cost about $490. If this feature were sold as a flat add-on, the heavy tail would decide your margin.

A background version of the same idea: if the product also tags and summarizes every ticket overnight for analytics, running those 500,000 short Haiku 4.5 calls through the Batch API halves that line, from about $1,250 to about $625 per month under the same pricing [2].

The cost levers, ranked

Ordered by how often they deliver meaningful savings relative to effort, based on our production work:

Right-size the model per task. Most features mix easy and hard requests. Routing easy ones to cheaper models is the largest structural saving available. Academic work supports this: the RouteLLM researchers showed learned routers that switch between a strong and a weak model can cut costs by more than 2x on public benchmarks without sacrificing response quality [9]. You do not need a trained router to start; rules based on input type or a small classifier often capture most of the benefit. Choosing candidates is covered in choosing an LLM for your product.

Prompt caching. Put everything static first (instructions, examples, tool definitions) and everything variable last, because caches match on exact prefixes [5][2]. Then verify it works: both OpenAI and Anthropic report cached token counts in the usage object, so track cache hit rate as a metric. A timestamp or user name near the top of a prompt can quietly drop your hit rate to zero.

Trim context. Send what the task needs. Improve retrieval precision so fewer, better chunks go in. Summarize or window long conversation histories. Only include the tool definitions relevant to the current step.

Control output and reasoning. Ask for structured or concise formats, set maximum output lengths, and use the lowest reasoning setting that passes your evals. Reserve reasoning models for tasks that measurably benefit.

Batch non-interactive work. Enrichment, tagging, summaries for reports, backfills, and eval runs rarely need real-time answers. Move them to batch APIs and stack caching on top where prompts share a prefix [2].

Application-level caching. If users repeatedly ask identical questions, cache complete responses by exact match. Semantic caching, which reuses answers for similar questions, can save more but carries real risk: slightly different questions can need different answers, cached content goes stale, and in multi-tenant products a cache key that ignores tenant and permissions becomes a data leak.

Self-hosting open models. Running open-weight models on your own GPUs can make sense at high, steady volume with an ML operations team to support it. For most products adding their first AI features, API pricing plus the levers above wins on total cost once you account for engineering time, idle capacity, and keeping up with model releases.

Monitor cost like a production metric

Cost regressions ship the same way bugs do: a prompt change adds 2,000 tokens, a new tool definition lands in every call, or a model upgrade changes token counts. Catch them with the same discipline.

• Log every call with tenant, feature, model, prompt version, and token counts split into uncached input, cached input, cache writes, output, and reasoning.
• Track cost per successful outcome, not just cost per call. A cheaper model that users regenerate three times is not cheaper.
• Watch cache hit rate per feature and alert when it drops.
• Set per-tenant and per-user budgets with hard limits. OWASP includes unbounded consumption among its top ten risks for LLM applications, covering both runaway costs and abuse [10].
• Run cost checks in CI alongside quality evals, so a prompt change that doubles tokens fails before it merges.

Budgets and rate limits also double as reliability controls, which we cover in making LLM features reliable.

Price the feature so margins survive heavy users

Your cost model should shape your pricing, not the other way around. The common options:

Whatever you choose, price against the cost of the action at your realistic heavy-user volume, and leave headroom for the reasoning, context, and model upgrades you will want to add later.

Pre-launch cost checklist

• Every model call behind each user action is mapped and counted.
• Token counts come from real inputs, split by input, cached, output, and reasoning.
• Regeneration, retry, and conversation-length multipliers are in the model.
• Cost is modeled per tenant, including the heaviest realistic accounts.
• Prompts put static content first, and cache hit rate is tracked.
• Each task uses the smallest model that passes evals.
• Non-interactive workloads run through batch APIs.
• Per-tenant budgets, hard limits, and cost alerts are live.
• Pricing covers heavy-user cost with margin to spare.

Keep reading

• How to add AI features to an existing software product (the pillar for this series)
• Choosing an LLM for your product
• Azure OpenAI vs OpenAI API vs AWS Bedrock for enterprise apps
• Latency, rate limits, and fallbacks: making LLM features reliable
• What it costs to build an enterprise AI assistant

CMW AI Labs builds production AI features with cost engineering designed in from the first prototype: call-graph mapping, caching, model routing, and per-tenant metering. If your AI feature’s bill is growing faster than its adoption, talk to our team.

Sources

1. IT Brief UK, “Gartner forecasts AI spending to hit USD $64bn in 2026,” July 21, 2026. https://itbrief.co.uk/story/gartner-forecasts-ai-spending-to-hit-usd-64bn-in-2026
2. Anthropic, “Pricing,” Claude Platform Docs (accessed September 2026). https://platform.claude.com/docs/en/about-claude/pricing
3. OpenAI, “Reasoning models,” API documentation. https://developers.openai.com/api/docs/guides/reasoning
4. Microsoft Learn, “Azure OpenAI reasoning models.” https://learn.microsoft.com/en-us/azure/foundry/openai/how-to/reasoning
5. OpenAI, “Prompt caching,” API documentation. https://developers.openai.com/api/docs/guides/prompt-caching
6. OpenAI, “Batch API,” API documentation. https://platform.openai.com/docs/guides/batch
7. Google, “Batch Mode,” Gemini API documentation. https://ai.google.dev/gemini-api/docs/batch-mode
8. Epoch AI, “LLM inference prices have fallen rapidly but unequally across tasks.” https://epoch.ai/data-insights/llm-inference-price-trends
9. Ong et al., “RouteLLM: Learning to Route LLMs with Preference Data,” ICLR 2025. https://arxiv.org/abs/2406.18665
10. OWASP GenAI Security Project, “OWASP Top 10 for LLM Applications 2025.” https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/