Monitoring LLM Quality in Production

A feature that passes every pre-launch eval can still degrade silently. Your eval set covers the cases you thought of; production supplies the rest. Meanwhile the ground shifts underneath: customers upload document formats you never tested, new tenants bring vocabulary your retrieval was not tuned for, someone edits a prompt, and providers periodically update the checkpoints behind a model name without changing the name [1].

None of that produces an error. Latency stays flat, error rates stay clean, and the dashboards stay green while the answers get worse.

This is the last article in our series on evals and LLMOps, and it covers the layer that runs forever: what to measure in production, how to alert without drowning in false alarms, and how production failures feed back into the eval set that catches them next time.

Four layers of production signal

Each layer answers a different question, and skipping any of them leaves a blind spot.

Most teams build the first two and stop, because those are what conventional monitoring gives you for free. The third and fourth are where AI features actually fail. Note also that provider dashboards cover none of this well: they show aggregate token usage, not per-tenant or per-feature breakdowns, quality scores, or the correlation between a prompt version and a quality change [1].

Run evaluations on live traffic

Offline evals run on a fixed dataset. Online evals run your validated judges against a sample of real production traffic, which is the only way to measure quality on inputs nobody anticipated.

Sample a share, not everything. Judge calls cost money and add load, so a modest percentage of traffic is the usual starting point, with published guidance commonly in the range of 5% to 10% [2]. Treat that as a starting point rather than a standard: tune it against your traffic volume, your judge cost, and how quickly you need to detect a change.

Stratify the sample. Uniform random sampling under-represents exactly the segments most likely to be broken. Sample proportionally across tenants, document types, languages, and feature paths so a small customer’s collapse is visible.

Always keep a purely random slice. If you only judge traces that some other signal already flagged, your quality metric will confirm what you already knew and never surface anything new.

Use the same validated judges as offline. A different judge in production means production numbers and CI numbers are not comparable, which defeats the purpose. And judge error rates still apply: report them alongside the scores. See validating an LLM judge.

For retrieval-grounded features, faithfulness and context relevance are the metrics to run continuously, because they catch the failure mode that looks fine to a user and is not. Groundedness failures are common enough at frontier quality to be worth watching rather than assuming away [2]. See RAG evaluation.

Watch the silent failures

These are the degradations that produce no error and that infrastructure monitoring cannot see.

Refusal rate. A model that quietly begins declining a few percent of valid questions after a provider updates a checkpoint is arguably worse than an outage, because nothing alerts and users simply conclude the feature is useless [3]. Track refusals as a first-class metric, split by whether refusal was correct.

Truncation. Finish reasons that shift toward length limits mean answers are being cut off. Users report this as the AI getting lazy; your logs report it as success.

Schema and guardrail failures. Structured output that stops parsing, or content filters firing more often, both indicate something upstream changed.

Cache hit rate. A prompt edit that moves a variable above the static prefix can silently destroy caching, raising cost and time to first token together [4]. See LLM API costs in production.

Output length and format drift. Median response length is a cheap proxy that catches a surprising number of prompt and model changes.

Latency distribution shifts. Not the mean. A p99 that doubles while the median holds is a real degradation for the tail of users experiencing it.

Detect drift on inputs, not just outputs

Input drift is what eventually breaks a feature that was working. New customer segments, new document formats, new languages, and seasonal shifts in what people ask all change the distribution your evals were built against.

One important difference from classic ML monitoring: the drift surface for LLM features is wider and faster-moving than for tabular models, and it generally needs evaluator-based detection rather than statistical distribution tests on features [3]. Practical signals that work:

• New input categories: cluster incoming requests periodically and watch for clusters your eval set does not cover.
• Retrieval health: a rising share of queries where top-k similarity scores are low often precedes answer quality dropping.
• Corpus change: documents added, removed, or reprocessed change retrieval behavior even when nothing in your code changed.
• Tenant onboarding: a large new customer is a distribution shift with a date attached, and their first weeks deserve closer watching.

Use the behavior you already collect

If your interface captures accept, edit, discard, and regenerate, you have a continuous quality signal that costs nothing and comes from real users doing real work. Rising regeneration rates and rising edit distance both indicate quality problems before anyone files a ticket. Escalation to a human, where your product offers it, is the clearest signal of all.

These signals are also the best sampling input for review: traces that were discarded or heavily edited are the ones worth reading. See AI copilot UX patterns and error analysis for LLM failures.

Alert on joint conditions, not single metrics

Single-threshold alerts on global averages produce noise, and noise produces engineers who ignore alerts. Combining conditions is what makes an alert worth paging on: a quality metric below threshold, and a runtime signal moving, and a specific cohort affected. Practitioner reporting suggests joint-condition alerts reduce false positives substantially compared with single-metric alerts [3].

Two design rules that matter as much as the thresholds:

Every alert needs an owner and a next action. A threshold that fires into a Slack channel nobody owns is theater [3]. The next action should be named in advance: roll back the prompt, fail over to the fallback model, tighten a guardrail, or repair the dataset.

Route by what triggered it. An alert on a faithfulness drop should reach whoever owns that evaluator or that prompt, not a general on-call rotation with no context.

A reasonable starting set of alerts:

Note that thresholds should be derived from your own observed variance rather than copied from an article. The same statistical reasoning that governs CI gates applies here: a change smaller than your measurement noise is not a signal. See regression testing AI features in CI.

Segment everything, always

Aggregate quality metrics hide segment collapse as a matter of routine. A feature at 92% overall can be at 45% for one tenant, one document type, or one language, and the aggregate will never show it.

Segment by tenant, feature path, document format, language, user cohort, model, and prompt version. That last one is not optional: without a prompt version on every span, A/B rollouts and post-incident attribution become guesswork [3]. See LLM observability and tracing.

Per-tenant quality is also commercially useful. Knowing which accounts are getting the worst results tells your customer success team where churn risk is building, well before a renewal conversation.

Close the loop

Monitoring that does not feed back into evaluation is just a nicer way to watch things break.

The loop: production monitoring flags a failure, a human reviews the trace, the failure is categorized, and the case is added to the eval set so CI catches its recurrence. That is what keeps an eval suite representative rather than slowly becoming a museum of last year’s bugs.

Alongside the automated signal, keep the human cadence from error analysis: a full review pass on fresh traces every few weeks, and a smaller weekly sample in between, weighted toward outliers and flagged traces. New systems need this weekly until failure patterns stabilize; mature ones can stretch. Always run one after an incident.

Treat quality incidents as incidents

Quality degradations deserve the same discipline as outages, and most teams have no process for them at all.

• Declare them. A significant quality regression is an incident even when every system is technically healthy.
• Attribute by version first. Prompt version, model version, application version, corpus version. Most quality incidents are a change somebody shipped.
• Roll back the same way. Prompts should be revertable independently of application deploys.
• Write the postmortem, including why existing evals missed it, and add the cases that would have caught it.

Some regulated contexts also require you to evidence ongoing monitoring rather than just perform it. See enterprise AI governance and our EU AI Act guide.

Common mistakes

• Monitoring only infrastructure and cost, so quality degrades invisibly.
• Sampling for online evals only from traces another signal already flagged.
• Reporting aggregate quality with no segmentation.
• Single-metric threshold alerts, which train everyone to ignore alerts.
• Alerts with no named owner and no predefined action.
• Missing prompt version on spans, making attribution guesswork.
• Ignoring refusal rate, truncation, and cache hit rate.
• Never adding production failures back into the eval set.
• Assuming a model behind a stable name behaves identically over time.

Checklist

• Infrastructure, cost, quality, and behavior are all monitored.
• Validated judges run on a stratified sample of live traffic, including a random slice.
• Refusal rate, truncation, schema failures, and cache hit rate are tracked.
• Input drift is monitored, not just output quality.
• User behavior signals feed both monitoring and trace sampling.
• Alerts use joint conditions, with an owner and a predefined action each.
• Every metric is segmented, including by prompt version.
• Production failures are categorized and added to the eval set.
• Quality regressions follow an incident process with postmortems.

Keep reading

• LLM evals and LLMOps: how to know your AI feature actually works (the pillar for this series)
• Error analysis for LLM failures
• How to validate an LLM judge
• RAG evaluation: measuring retrieval and generation separately
• LLM observability and tracing
• Regression testing AI features in CI

CMW AI Labs builds production monitoring that watches quality, not just uptime: online evaluation on live traffic, segmented dashboards, and a loop that turns every production failure into a regression test. If your AI feature has been live for months and nobody can say whether it still works, talk to our team.

Sources

1. ValueStream AI, “AI Monitoring in Production 2026: LLM Observability and Drift Detection,” May 2026. https://valuestreamai.com/blog/ai-monitoring-in-production-guide-2026
2. OpenObserve, “LLM Monitoring Best Practices: Complete Guide for 2026,” April 2026. https://openobserve.ai/blog/llm-monitoring-best-practices/
3. Future AGI, “What Is LLM Monitoring?” May 2026. https://futureagi.com/glossary/llm-monitoring/
4. Anthropic, “Pricing,” Claude Platform Docs (accessed September 2026). https://platform.claude.com/docs/en/about-claude/pricing