contextjet-ai/awesome-llm-observability/debug-agent-from-traces

Debug an agent from its trace

A trace is a tree of spans (parent request → LLM/tool/retrieval children). Most agent failures are visible in it if you read it in the right order. Don't guess from the final output - walk the tree.

Triage by symptom

The systematic walk

1. Start at the root, confirm the user input is what you expect.
2. Follow to the first LLM call. Read the actual rendered prompt + context (not the template). Most "model is dumb" bugs are actually "we fed it the wrong context".
3. Check each tool/retrieval span in order: inputs correct? output sane? error?
4. Find the divergence point - the first span where reality differs from intent. Fix there, not at the output.
5. Check token + latency on every LLM span to catch cost/perf issues even when the answer is right.

Common root causes (in order of frequency)

• Wrong/empty retrieved context → the LLM was set up to fail. (Retrieval bug.)
• Prompt/template rendering bug → a variable didn't interpolate; context is blank or duplicated.
• Swallowed exception → a try/except that logs and returns empty, dropping context or output silently. Grep for these.
• Retry storm → the same call repeated N times (rate limit / transient error) inflating cost + latency.
• Context bloat → the whole history re-sent each turn; input_tokens grows every step.

Turn the fix into a regression test

Once you find the divergence, capture that input as an eval case (see the add-llm-evals skill) so the bug can't come back. Debugging a trace and not adding a test means you'll debug it again next month.

Anti-patterns

• Re-prompting the model when the real bug is upstream retrieval/rendering.
• Reading only the final output and inferring the cause.
• Fixing the symptom without adding a regression test.