Topic hub
How to tell whether an AI feature actually works: reading real failures, building test sets from production, validating the judge that grades them, and keeping the measurement running after launch.
Start with the pillar guide
Read a hundred real traces and categorize the failures before you design a single metric. Generic helpfulness and coherence scores diagnose nothing, because your failures are specific. Build the test set from production traffic, validate any LLM judge against human labels the way you would a classifier, and run the suite in CI on every prompt and model change.
Read the guideHow to evaluate a retrieval-augmented system so failures point at a cause: IR metrics for retrieval, the six question-context-answer relationships for generation, building test sets without manual annotation, and a symptom-to-fix diagnosis table.
Passing evals prove only that the feature worked on the cases you thought of. In production, run your validated judges on a stratified sample of live traffic, track the silent failures infrastructure monitoring misses (refusals, truncation, cache hit rate), and alert only on joint conditions with a named owner. Then feed every failure back into the eval set.
A CI gate has to be cheap, fast, and statistically meaningful, and most designs manage two of the three. Split it instead: deterministic assertions and critical-slice invariants block every pull request, while the judge suite runs nightly and gates on a paired comparison rather than a raw threshold. Size the eval set for the regression you actually need to catch.
Capture the whole tree for every interaction, not just the final call: each model call, retrieval and tool invocation, with token counts, finish reason, retry and fallback state, and tenant, feature and prompt version on every span. Use OpenTelemetry's gen_ai names rather than inventing your own, and treat content capture as a data-governance decision.
Read your own system's traces before you build a single metric. Sample about a hundred, write free-text notes on the first thirty yourself, then group those notes into specific named failure categories and count them. The ranked list tells you which evaluators are worth building, and which failures a one-line prompt fix would remove.
Using an LLM to grade outputs is easy; knowing whether the grader is right is the hard part. Have one domain expert label real outputs pass or fail with written critiques, build the judge prompt out of those critiques, then hold out a test set and report true positive and true negative rates rather than raw agreement.
