LLM Observability and Tracing

Everything else in evaluation depends on this one. You cannot run error analysis on failures you cannot reconstruct, cannot validate a judge without the inputs it judged, and cannot diagnose a production incident from a 200 status code and a latency number.

Tracing is the data plane for every other practice in this series, and it should be in place before the first prototype leaves anyone’s laptop. This guide covers what to capture, how to structure it, which standard to follow, and where telemetry stops being enough. It is part of our series on evals and LLMOps.

Why standard APM is not enough

Traditional application monitoring assumes deterministic code paths: the same input produces the same output, so a stack trace and a request log are usually sufficient to reproduce a bug. LLM features break that assumption in several ways at once [1].

Non-determinism. The same prompt can produce different outputs. Reproducing an issue requires the exact input, the model parameters, and the temperature in effect at the time, because re-running the request will not reliably reproduce it [1].

A token-based cost model. Latency and cost track token counts rather than request counts, so request-rate dashboards tell you almost nothing about spend [1]. See LLM API costs in production.

Failures with no error. A model can return a fluent, well-formed, completely unsupported answer in 400 milliseconds with a 200 status code. Nothing in conventional telemetry registers a problem.

Multi-step execution. One user action can trigger retrieval, several model calls, and a chain of tool invocations. A flat request log flattens exactly the structure you need to see.

Structure traces as trees, not lines

A trace is the complete record of one user interaction. For anything beyond a single call, it should be a tree: a root span for the user-facing operation, with child spans for each model call, retrieval, and tool invocation beneath it, nested to reflect actual control flow [1].

For an agent loop, that means a span per iteration, with the model call and each tool execution as children of the iteration that triggered them. The shape of the tree is itself diagnostic. A trace with fourteen iterations where you expected three tells you something before you read a single attribute.

Capture the whole tree even when the final answer looks fine. A correct answer reached through a broken path is a bug that has not surfaced yet.

What to capture on every model call

Two fields earn their place more than they look. Finish reason separates a clean stop from a truncated response from a tool call, and truncation silently degrades quality in ways users report as “the AI is being lazy.” Retry and fallback counts move before your error rate does during an incident, which makes them the leading indicators worth alerting on. See making LLM features reliable.

The correlation row is what makes segmented analysis possible later. Every eval practice in this series depends on being able to slice results by tenant, document type, feature, and user cohort, and you cannot add those dimensions retroactively to traces already collected.

Use the standard vocabulary

Do not invent attribute names. OpenTelemetry’s GenAI semantic conventions define a shared gen_ai.* namespace so that telemetry means the same thing regardless of which library, language, or provider produced it [2].

The core attributes cover what you would expect: the request model, input and output token usage, and the reason generation finished [3]. When an application is configured to record content, messages and tool calls are captured as structured attributes for system instructions, input messages, and output messages [3]. The conventions also define metrics for client, server, workflow, agent, and tool operations, plus provider-specific conventions for systems including OpenAI, Anthropic, Azure AI Inference, and AWS Bedrock [4].

Three practical cautions.

Stability varies by attribute. Every attribute carries a stability marker under OpenTelemetry’s standard three-tier system: Development, Release Candidate, and Stable. Core chat and embedding attributes are solid enough to build production dashboards on; agent and tool-orchestration conventions are still settling and should be treated as provisional [2]. The dedicated GenAI conventions repository is itself still in Development and evolving independently [4].

Names have already migrated. You will encounter spans carrying both older and newer provider-identifying attributes side by side during the transition [2]. Pin the convention version you emit rather than assuming stability.

Adoption is directional, not automatic. Shared semantics reduce re-instrumentation and translation work, but backend support, convention versions, custom attributes, and your own queries all still matter [4]. Standardizing does not make telemetry portable by itself; it makes it portable with less work.

Where you need fields the spec does not cover, such as cost in your currency, tenant identifiers, or cache behavior, add custom attributes as extensions alongside the standard ones rather than replacing them [4].

Treat content capture as a data-governance decision

This is where instrumentation meets your security review, and where the defaults are deliberately conservative.

Content capture is opt-in by design. The detailed inference-content event is marked opt-in in the conventions, and implementation support varies by language [4]. Instrumentation libraries commonly default content capture to off for privacy, with explicit configuration required to turn it on. OpenTelemetry provides a standard environment variable for controlling message content capture, which lets one setting govern behavior across services regardless of what each individual config says [5].

Prefer events over attributes for content. Storing full prompt text in span attributes is an anti-pattern: attributes are always indexed, carry size limits, and push sensitive text into your observability backend. Capturing content as span events instead allows it to be filtered or dropped at the Collector, without touching application code [1].

Redact before export, not after. Apply minimization and filtering in the pipeline, then enforce retention and access controls in the backend. An attribute name in a specification is not a data-governance policy [4].

Plan for the multi-tenant case. Prompts and retrieved context routinely contain your customers’ data. Decide who can read traces, how long they are retained, whether content is captured per tenant or globally, and what your contracts and your customers’ security questionnaires require. Full traces with content are frequently the right choice in development and a compliance problem in production. See enterprise RAG security and governance and enterprise AI governance.

A workable default: structural telemetry always on, content capture on in development and staging, on in production only for sampled traffic, under redaction, with short retention and restricted access.

Correlate multi-turn sessions

Single-call tracing is not enough for conversational features. A dedicated conversation identifier ties spans from separate turns into one traceable session [2], which is what lets you answer questions like whether quality degrades as conversations lengthen, or where in a ten-turn exchange the model lost the user’s stated constraint.

Add prompt version and application version alongside it. When quality drops, the first question is always what changed, and a version attribute answers it in a query rather than an archaeology session through deploy logs.

Carry evaluation results in the same telemetry

Evaluation scores belong attached to the operations that produced them. The conventions provide a standard carrier for evaluator scores and explanations, which means a judge’s verdict can travel with the span it graded [4].

This is worth setting up properly, because it turns two separate systems into one. A single query can then answer questions that otherwise require joining exports by hand: which tenants see the most faithfulness failures, whether failures cluster at high token counts, whether a specific document type correlates with low context relevance. See RAG evaluation.

Know where telemetry stops

OpenTelemetry represents evaluation results; it does not perform evaluation [4]. This boundary is the most common misunderstanding in the space.

A span can tell you a call returned 1,200 tokens in 850 milliseconds with a clean finish reason. It cannot tell you those tokens contradicted the retrieved sources [6]. Telemetry is the data plane. Quality measurement is a separate layer that reads that data and applies assertions, judges, and human review on top.

Teams that conflate the two end up with excellent dashboards showing latency, cost, and error rates, and no idea whether the feature is producing correct answers. The full stack looks like this:

Build a viewer people will actually use

Raw traces are JSON, and asking a domain expert to annotate raw JSON guarantees shallow annotation. Content attributes in particular are large, and many platforms render them as unreadable JSON blobs [3].

A purpose-built review interface is one of the highest-leverage things you can build here, and modern coding assistants make it a few hours of work. What matters: render content the way the domain expects it, show the entire interaction plus the downstream outcome on one screen, collapse the detail nobody reads, and support filtering and search so a reviewer can chase a hypothesis. More on this in error analysis for LLM failures.

Sample deliberately

Full-fidelity tracing with content on every request is expensive in storage and, in some architectures, in latency. Some sensible defaults:

• Structural telemetry on 100% of traffic. Tokens, latency, cost, and outcomes are small and are what your alerts read.
• Content on a sample in production, weighted toward the traces you are most likely to want: errors, retries, fallbacks, unusually long sessions, low judge scores, and user discards or regenerations.
• Always keep some uniformly random traces, or your sample will only ever contain the failures your existing signals already detect.
• Full capture in development and staging, where the privacy calculus is different.

Common mistakes

• Instrumenting after launch, so the first production failures cannot be reconstructed.
• Logging only the final output, discarding retrieved context and intermediate steps.
• Inventing custom attribute names instead of using the standard vocabulary.
• Putting full prompt content in indexed span attributes.
• Enabling content capture in production without redaction, retention limits, or access controls.
• Omitting tenant, feature, and version identifiers, making segmentation impossible later.
• Assuming telemetry answers quality questions.
• Sampling only flagged traces, so the sample confirms what you already knew.

Checklist

• Traces are trees, capturing every model call, retrieval, and tool invocation.
• Token counts are split by category, including cached and reasoning tokens.
• Finish reason, retry count, and fallback usage are recorded.
• Conversation, tenant, user, feature, and prompt version are on every span.
• Attribute names follow the GenAI conventions, with the version pinned.
• Content capture is opt-in, redacted before export, with retention and access controls.
• Evaluation results are attached to the spans that produced them.
• A usable trace viewer exists for non-engineers.
• Structural telemetry covers all traffic; content sampling includes random traces.

Keep reading

• LLM evals and LLMOps: how to know your AI feature actually works (the pillar for this series)
• Error analysis for LLM failures
• How to validate an LLM judge
• RAG evaluation: measuring retrieval and generation separately
• Monitoring LLM quality in production

CMW AI Labs instruments AI features before building them, so error analysis, evals, and production monitoring all have the data they need from day one. If your AI feature is live and you cannot reconstruct a failure, talk to our team.

Sources

1. Uptrace, “OpenTelemetry for AI Systems: LLM and Agent Observability,” April 2026. https://uptrace.dev/blog/opentelemetry-ai-systems
2. OpenObserve, “OpenTelemetry GenAI Semantic Conventions: A Practical Guide,” August 2026. https://openobserve.ai/blog/opentelemetry-genai-semantic-conventions/
3. OpenTelemetry, “Inside the LLM Call: GenAI Observability with OpenTelemetry,” May 2026. https://opentelemetry.io/blog/2026/genai-observability/
4. TrueFoundry, “OpenTelemetry GenAI Semantic Conventions,” January 2026. https://www.truefoundry.com/blog/opentelemetry-genai-semantic-conventions
5. NVIDIA, “Capturing Prompt and Response Content,” NeMo Guardrails documentation. https://docs.nvidia.com/nemo/guardrails/observability/tracing/content-capture
6. Fiddler AI, “OpenTelemetry for AI Observability: What It Covers and Where It Stops,” July 2026. https://www.fiddler.ai/blog/opentelemetry-ai-observability-guide