How to Run Error Analysis on LLM Failures

Most teams building an AI feature skip straight to the part that feels like engineering: pick metrics, wire up a judge, build a dashboard. The dashboard then reports helpfulness at 4.2 and hallucination at 0.08, and nobody can act on either number, because neither was derived from anything the system actually does wrong.

Error analysis is the step that should come first. It is a structured read of your own system’s outputs, producing a counted list of the specific ways your application fails. Practitioners who teach this material to thousands of engineers call it the most important activity in evals, on the grounds that it determines which evals are worth writing at all [1]. The same source reports spending 60% to 80% of development time on error analysis and evaluation rather than on building automated checks [1].

It is also cheap. A first pass costs a few days and needs no infrastructure beyond a way to read traces.

This guide walks the method end to end. It is part of our series on evals and LLMOps.

What you are producing

The output of error analysis is a failure taxonomy: a list of specific, named ways your application fails, with a count for each. Something like:

(Illustrative counts, not client data.)

That table is worth more than any dashboard, because every row is specific enough to fix and counted well enough to prioritize. It tells you what to build evaluators for, and in what order.

Step 1: Get traces in front of you

A trace is the complete record of one user session: the input, every retrieval, every tool call, every intermediate model response, and the final output [1]. You need the whole thing, because a bad answer is frequently caused by something three steps upstream.

If you do not have tracing yet, add it before anything else. See LLM observability and tracing.

If you do not have production traffic yet, generate synthetic traces, but do it with structure. Prompting a model for fifty test queries yields generic, repetitive output. The better approach is to define the dimensions that describe real variation in your users’ inputs (issue type, document format, customer mood, language, query complexity), write twenty combinations by hand to understand the space, then scale up by generating more combinations and converting each into a natural-language query in a separate step [1]. Run those through your actual system to get full traces. Synthetic data cannot tell you how often a failure occurs in production, and it tends to miss the quirks of specialized domains, so replace it with real traffic as soon as you have any [1].

Step 2: Sample deliberately

How you choose traces determines what you find. The main options, from most exploratory to most targeted [1]:

Start exploratory and move toward targeted as you learn. Critically, keep some random traces in every batch, or you will only ever find the failures your current signals already describe [1].

Step 3: Open coding, and do it yourself

Read traces one at a time and write a short free-text note about anything wrong from the user’s point of view. No categories yet, no rubric, no scores. This is adapted from qualitative research, where it is called open coding, and the closest everyday description is journaling [1].

Four rules make this work:

Do at least 30 yourself before letting an assistant suggest anything. If a model starts proposing failure categories early, its guesses anchor your judgment and you stop noticing the problems that depend on product context [1].

Note the first failure in each trace. Upstream errors cascade, so a trace often contains five symptoms of one cause. Once you have practice, you can tag independent failures too [1].

Have a domain expert do it. The person who understands your users should be reading the traces. For most teams one internal expert acting as the final authority on quality works better than a committee, because it eliminates annotation conflicts and decision paralysis [1]. If you genuinely need several experts to judge a single interaction, your product scope is probably too broad [1].

Do not outsource it. The point of error analysis is building product intuition, and that does not transfer through a deliverable. External annotators without domain context produce superficial labels and often create more disagreement than they resolve [1]. Bringing in outside subject-matter experts to act as your internal experts is a different thing and is fine.

Expect your standards to shift as you read, and do not treat that as a mistake. Researchers studying LLM evaluation documented the effect and named it criteria drift: people need criteria to grade outputs, but grading outputs is how they work out their criteria [2]. Plan on revisiting earlier notes.

Step 4: Axial coding into a taxonomy

Now group the notes. Cluster similar observations into distinct failure categories and count how many traces fall into each [1]. This is the step that converts a pile of notes into a prioritization map, and it is where the method earns its keep.

A model can help propose groupings once you have your first 30 to 50 notes written, but the clusters need your review [1]. The most common mistake is accepting vague categories. “Temporal issues” is not an actionable category; if you handed it to someone else to label with, they would not know what you meant. “Date formatting error” is [3]. The test: could a second person apply this category consistently without asking you what it means?

Watch for the opposite failure too. A model asked to group notes may fold “crashes after login” and “login is slow” into a single login-issues bucket, even though one is a stability bug and the other a performance problem requiring a completely different fix [1].

Then count. A pivot table over your annotations turns messy notes into a ranked list of problems, and that ranking is the entire point: you stop guessing what to fix and start seeing which failures recur and which are riskiest [4].

Step 5: Know when to stop

Keep reviewing until new traces stop revealing new failure modes or changing the ones you have. Qualitative researchers call this theoretical saturation. A working guideline is at least 100 traces, and more while you are still learning [1].

The sequence that works well in practice: sample a diverse pool of roughly 100, annotate the first 30 yourself, then have an assistant search the rest for instances of the failures you described, reviewing and correcting every suggestion. That keeps a human in charge of the criteria while removing the tedium of reading all 100 sequentially [1].

Development time practitioners report spending on error analysis and evaluation

60-80%

Rather than on building the automated checks themselves.

Traces to annotate yourself first

30

Before letting an assistant propose any category, so its guesses cannot anchor yours.

Working minimum for a first pass

100

Then stop when fresh traces stop revealing new failure modes.

Step 6: Decide what to do with each failure mode

Not every failure needs an evaluator. Many are just bugs, and the fastest path is to fix them and move on.

In particular, check first whether the failure is something you never actually asked for. Teams routinely discover the model is not meeting a preference nobody wrote down: response length, formatting, a required disclaimer, a step-by-step structure. Fix the prompt rather than building infrastructure to measure a gap you can close in a line [1].

For the rest, the decision is a cost-benefit call:

Code assertions are cheap: JSON parses, the tool call carried the right arguments, the citation exists, the currency is formatted correctly. Judges are expensive, typically needing 100 to 200 labeled examples per failure mode plus ongoing maintenance, so reserve them for failures you will iterate on repeatedly [1]. Whichever you build, use binary pass or fail rather than a 1-to-5 scale: adjacent points on a Likert scale mean different things to different annotators, detecting differences needs larger samples, and annotators drift toward the middle to avoid making a call [1].

Validating a judge is its own discipline, covered in validating an LLM judge.

Special cases

Multi-turn conversations. Start with a single pass/fail on whether the conversation met the user’s goal, read the user-visible parts first, and only then dig into tool calls. When you find a failure, try to reproduce it as a single turn. If a shopping assistant gives the wrong return policy on turn four, ask the same question cold. If it still fails, the problem is retrieval or knowledge, not conversation context, and it is far easier to debug [1].

Agents and multi-step workflows. Evaluate end-to-end task success first, then add step-level diagnostics: tool choice, parameter extraction, error handling, context retention, and efficiency [1]. A transition failure matrix, with rows for the last successful state and columns for where the first failure occurred, shows where failures cluster and tells you which transition to investigate first [1]. See AI agents in the enterprise.

Retrieval-augmented features. Separate retrieval failures from generation failures before you categorize anything else, because they have entirely different fixes [1]. See RAG evaluation and our enterprise RAG guide.

Make the review itself fast

Friction in the annotation interface shows up directly as worse annotation. A purpose-built review tool, which modern coding assistants make a few hours’ work, is one of the highest-leverage investments in the whole process [1]. What matters:

• Render traces the way the domain expects. Emails should look like emails, code should be syntax-highlighted, and less important detail should be collapsed rather than dumped as raw JSON [1].
• Show everything on one screen, including the downstream outcome that proves the task succeeded, so a non-technical reviewer can judge whether the appointment was actually booked rather than whether a tool call returned 200 [1].
• Add progress indicators and keyboard shortcuts. Bounded sessions and hotkeys keep reviewers in flow [1].
• Support filtering, search, and clustering so reviewers can chase a hypothesis.

Keep it running

Error analysis is not a one-time project. A reasonable cadence: re-run a full pass on 100-plus fresh traces every two to four weeks, or whenever you ship a significant change such as a new feature, prompt rewrite, or model switch. Between passes, review 10 to 20 traces weekly, weighted toward outliers like unusually long sessions, repeated retries, or anything automated monitoring flagged [1]. New systems need this weekly until failure patterns stabilize; mature ones can stretch to monthly. Always run one after an incident or a spike in complaints [1].

The behavioral signals your interface already produces are a good sampling input here, since edits, discards, and regenerations point at traces worth reading. See AI copilot UX patterns and monitoring LLM quality in production.

Using the results to get buy-in

If you need to convince a team to invest in evaluation, do not argue for evals in the abstract. Do the analysis first and show them what you found: the top failure modes, how often the costly ones occur, and the surprising things users are doing with the product. A concrete report of issues caught before customers saw them lands better than a methodology pitch [1].

One caution on how you report progress afterward. A suite you pass 100% of the time is probably not testing anything; a pass rate closer to 70% often indicates evals that are actually stressing the system [1]. Optimize for catching real problems, not for a green dashboard.

Common mistakes

• Building judges and dashboards before reading a single trace.
• Letting a model do the open coding, so you never develop intuition for your own failures.
• Vague axial codes nobody else could apply consistently.
• Tagging every symptom in a trace instead of the first upstream failure.
• Outsourcing annotation to people without domain context.
• Reviewing only traces users complained about.
• Building an expensive evaluator for a failure a one-line prompt fix would eliminate.
• Running error analysis once, before launch, and never again.

Checklist

• Full traces are captured and readable in one place.
• The sample is diverse, with some random traces included.
• A domain expert wrote free-text notes on at least 30 traces personally.
• Notes are grouped into specific, unambiguous failure categories.
• Each category is counted, and the list is ranked.
• Obvious prompt gaps were fixed before any evaluator was built.
• Each remaining failure mode has a decision: assertion, judge, or accept.
• A re-run cadence is scheduled, not improvised.

Keep reading

• LLM evals and LLMOps: how to know your AI feature actually works (the pillar for this series)
• Validating an LLM judge
• RAG evaluation: retrieval and generation
• LLM observability and tracing
• Regression testing AI features in CI
• Monitoring LLM quality in production

CMW AI Labs runs error analysis as the first step on every AI engagement, then builds the evaluators the taxonomy justifies. If your AI feature is underperforming and you cannot say precisely how, talk to our team.

Sources

1. Hamel Husain and Shreya Shankar, “AI Evals: Everything You Need to Know,” May 2025, updated September 2026. https://hamel.dev/blog/posts/evals-faq/
2. Shankar et al., “Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences,” UIST 2024, arXiv:2404.12272. https://arxiv.org/abs/2404.12272
3. Aakash Gupta, “AI Evals Step-by-Step: Hamel Husain and Shreya Shankar Masterclass,” May 2026. https://www.aakashg.com/hamel-shreya-ai-evals-step-by-step/
4. Lenny’s Podcast, “Why AI evals are the hottest new skill for product builders,” September 2025. https://podpulse.ai/podcast/lennys-podcast-product-growth-career/lennys-podcast-product-growth-career-why-ai-evals-are-the-hottest-new-skill-for-product-builders-hamel-husain-amp-shreya-shankar-creators-of-the-1-eval-course