opensourceagi/qwksearch-research-agent/agent-tracing

Agent Tracing CLI Guide

@lobechat/agent-tracing is a zero-config local dev tool that records agent execution snapshots to disk and provides a CLI to inspect them.

How It Works

In NODE_ENV=development, AgentRuntimeService.executeStep() automatically records each step to .agent-tracing/ as partial snapshots. When the operation completes, the partial is finalized into a complete ExecutionSnapshot JSON file.

Data flow: executeStep loop -> build StepPresentationData -> write partial snapshot to disk -> on completion, finalize to .agent-tracing/{timestamp}_{traceId}.json

Context engine capture: In RuntimeExecutors.ts, the call_llm executor calls ctx.tracingContextEngine(input, output) after serverMessagesEngine() processes messages. AgentRuntimeService.executeStep buffers the call per step and forwards it to OperationTraceRecorder.appendStep as the typed contextEngine field. CE flows through this side channel rather than the events array so its heavy payload (agentDocuments, systemRole, …) never enters the Redis state pipeline (LOBE-9110).

Package Location

packages/agent-tracing/
  src/
    types.ts          # ExecutionSnapshot, StepSnapshot, SnapshotSummary
    store/
      types.ts        # ISnapshotStore interface
      file-store.ts   # FileSnapshotStore (.agent-tracing/*.json)
    recorder/
      index.ts        # appendStepToPartial(), finalizeSnapshot()
    viewer/
      index.ts        # Terminal rendering: renderSnapshot, renderStepDetail, renderMessageDetail, renderSummaryTable, renderPayload, renderPayloadTools, renderMemory
    cli/
      index.ts        # CLI entry point (#!/usr/bin/env bun)
      inspect.ts      # Inspect command (default)
      partial.ts      # Partial snapshot commands (list, inspect, clean)
    index.ts          # Barrel exports

Data Storage

• Completed snapshots: .agent-tracing/{ISO-timestamp}_{traceId-short}.json
• Latest symlink: .agent-tracing/latest.json
• In-progress partials: .agent-tracing/_partial/{operationId}.json
• Downloaded remote snapshots: .agent-tracing/_remote/{operationId}.json
• FileSnapshotStore resolves from process.cwd() — run CLI from the repo root

Remote Traces (Production / Staging)

Server deployments also upload completed snapshots to object storage (zstd-compressed; the key is stored in agent_operations.trace_s3_key).

Preferred: `lh trace op`. The server resolves the key and signs the object for the caller's own scope, so a LobeHub login is the only requirement — no TRACING_BASE_URL, no bucket domain, and no SQL to turn a topic id into an operation id:

lh trace op list --topic tpc_xxx # operations of a topic, newest first, with a TRACE column
lh trace op inspect op_xxx_agt_xxx_tpc_xxx_xxxx
lh trace op inspect op_xxx_agt_xxx_tpc_xxx_xxxx -T # tool injection (enabledToolIds, manifests)

TRACE = — in list means no snapshot was recorded for that run (it predates trace upload, or upload was off). A recorded snapshot can still 404 in storage after its retention window.

Backend: the agentTrace lambda router (getSnapshotUrl / listOperations). Note it is blocked for restricted API keys, so these commands need a real session, not a scoped key.

Fallback: the standalone `agent-tracing` CLI. It has no LobeHub session, so it builds the object URL itself and needs the bucket's public domain configured:

• env var: TRACING_BASE_URL=https://<bucket-public-domain>/agent-traces
• or .agent-tracing/.env in the repo root with the same TRACING_BASE_URL=... line

The deployment-specific value is private to each deployment and intentionally not recorded in this repo. Find the operation id by hand first:

SELECT id, trace_s3_key FROM agent_operations WHERE topic_id = 'tpc_xxx';

Either way the snapshot is cached to .agent-tracing/_remote/<opId>.json, and every inspect flag works the same as for local traces.

Implementation: packages/agent-tracing/src/store/loadSnapshot.ts (resolution order: local store → _remote/ cache → injected resolveDownloadUrl → TRACING_BASE_URL) and store/remote-store.ts (URL built as {base}/{agentId}/{topicId}/{opId}.json.zst). Reading a compressed snapshot needs Node >= 22.15.

Goal Trajectories

A goal is one complete _goal_ execution the way an operation is one complete agent execution, so it gets the same trace format one level up: GoalTrajectory : AdvanceSnapshot mirrors ExecutionSnapshot : StepSnapshot. There is no table of advances, exactly as there is no agent_steps table — goal_traces holds one rollup row per goal plus the object key, and the detail lives in the object.

The leaves join back down: an advance records the operationIds it put in flight (on tick.effects[].operationId), so lh trace op inspect <opId> continues from where the goal trace stops.

agent-tracing goal               # list local goal trajectories
agent-tracing goal goal_xxx      # the run: triggers, outcomes, graph, gates, ops
agent-tracing goal goal_xxx -a 3 # one advance in full, with the frontier it ranked
agent-tracing goal goal_xxx -j   # raw JSON

What each tick records is the decision input, not just the result: the graph it read (as a delta against the previous tick), the budget it evaluated, the responsible task's state, and every eligible work node including the ones it passed over. Without the losers a trace cannot answer "why not that node".

Storage follows the operation switch, so a deployment that keeps one keeps the other:

• Completed: goal-traces/{goalId}.json.zst
• In progress: goal-traces/_partial/{goalId}.json.zst (an unfinished long-horizon goal is the normal thing to inspect)
• Dev: .goal-tracing/{goalId}.json

Replay. replayGoalAgainstCurrentCoordinator(trajectory) re-runs the real decideNextMove over the recorded inputs and reports {advanceSeq, tickIndex, field, recorded, replayed} wherever the current coordinator would now choose differently. This reproduces the coordinator's _decisions_ exactly; it says nothing about whether the dispatched work would have gone the same way.

Implementation: packages/agent-tracing/src/goal/, recorded through apps/server/src/services/goal/advanceGoal.ts (the single funnel every advance passes through) via the onDecision side channel on GoalService.tick.

CLI Commands

All commands run from the repo root:

# View latest trace (tree overview, `inspect` is the default command)
agent-tracing
agent-tracing inspect
agent-tracing inspect <traceId>
agent-tracing inspect latest
 
# List recent snapshots
agent-tracing list
agent-tracing list -l 20
 
# Inspect specific step (-s is short for --step)
agent-tracing inspect <traceId> -s 0
 
# View messages (-m is short for --messages)
agent-tracing inspect <traceId> -s 0 -m
 
# View full content of a specific message (by index shown in -m output)
agent-tracing inspect <traceId> -s 0 --msg 2
agent-tracing inspect <traceId> -s 0 --msg-input 1
 
# View tool call/result details (-t is short for --tools)
agent-tracing inspect <traceId> -s 1 -t
 
# View raw events (-e is short for --events)
agent-tracing inspect <traceId> -s 0 -e
 
# View runtime context (-c is short for --context)
agent-tracing inspect <traceId> -s 0 -c
 
# View context engine input overview (-p is short for --payload)
agent-tracing inspect <traceId> -p
agent-tracing inspect <traceId> -s 0 -p
 
# View available tools in payload (-T is short for --payload-tools)
agent-tracing inspect <traceId> -T
agent-tracing inspect <traceId> -s 0 -T
 
# View user memory (-M is short for --memory)
agent-tracing inspect <traceId> -M
agent-tracing inspect <traceId> -s 0 -M
 
# Raw JSON output (-j is short for --json)
agent-tracing inspect <traceId> -j
agent-tracing inspect <traceId> -s 0 -j
 
# List in-progress partial snapshots
agent-tracing partial list
 
# Inspect a partial (use `inspect` directly — all flags work with partial IDs)
agent-tracing inspect <partialOperationId>
agent-tracing inspect <partialOperationId> -T
agent-tracing inspect <partialOperationId> -p
 
# Clean up stale partial snapshots
agent-tracing partial clean
 
# Map the context window composition of every LLM call (cm / map are aliases)
agent-tracing ctx-map
agent-tracing ctx-map <operationId|traceId|path.json>
agent-tracing ctx-map --html            # standalone report under .agent-tracing/_reports/
agent-tracing ctx-map --html out.html
 
# Re-issue a recorded LLM call against other models (see "replay" below)
agent-tracing replay <operationId|traceId|path.json>
agent-tracing replay <target> -s 4 -m openai/gpt-5,anthropic/claude-sonnet-5
agent-tracing replay <target> --all-steps

replay — Re-issue a Frozen Call

A snapshot already freezes everything one LLM call saw: steps[].contextEngine.output is the exact message array sent to the model, context.payload.tools the toolset it could reach. replay sends that frozen payload back out with only the model swapped, so a difference in output is attributable to the model rather than to context assembly. If every model fails the same payload, the context is at fault; if some pass, it is model selection.

Available from both CLIs — agent-tracing replay (reads LOBEHUB_JWT) and lh trace op replay (uses the lh login session). Both need credentials because the call goes out through the LobeHub chat route.

# One call: defaults to the last call_llm step and the model the op ran on
lh trace op replay <operationId>
lh trace op replay <operationId> -s 4 -m openai/gpt-5,anthropic/claude-sonnet-5
lh trace op replay <operationId> --judge "answers with a concrete file path"
 
# Every call of the operation
lh trace op replay <operationId> --all-steps
lh trace op replay <operationId> --all-steps --concurrency 8

--all-steps answers the question the whole feature exists for: take a run that succeeded, put another model on it, and see whether that model also gets the job done. It ends in a PASS / FAIL verdict from an llm-rubric judge comparing the replayed outcome against the recorded one; pass --judge "<criteria>" to define success yourself instead of using the default rubric.

The judge scores the outcome, not the route. A model that solved the same problem by calling different tools has passed. The per-call tool comparison (toolSignature) is reported underneath as supporting evidence — where the run took another path — and never decides pass / fail. A final call that never reached the model is a FAIL, not a missing verdict.

Each call is replayed independently, against the payload the harness actually built for it, so a different answer at call 2 cannot contaminate call 4, a call that fails to reach the provider costs only itself, and the calls go out concurrently (4 at a time by default).

Chaining the nodes — feeding each replayed output into the next — was built and then removed. A trace cannot regenerate tool output, only hand back what was recorded, so the moment the model deviates there is no ground truth left; the run then measures nothing while still looking like it succeeded. Independent replay is the design, not a fallback.

Known limitation: sampling parameters are not recorded. The recorder stores the runtime step context under context.payload, not the provider request body, so temperature, top_p, max_tokens, penalties and reasoning config are absent from every trace written so far. Replays use the server's current defaults for those. Model-to-model comparisons stay valid (every target gets the same request); comparisons against the recorded output do not, for an operation that ran with non-default settings. --temperature / --max-tokens override explicitly. The CLI prints a note line whenever a replay is running without recorded parameters.

ctx-map — Context Window Composition