wibus-wee/cradle-app/cradle-observability-debugger

Observability Debugger

Use this skill when a chat turn behaves unexpectedly and you need concrete local evidence.

Data sources

• observability_events — structured error/warn/info events
• observability_incidents — deduplicated grouped events; TURN_STREAM_FAILED aggregates by code unless a producer supplies a narrower dedupe key
• backend_run_snapshots — Cradle-owned runtime-neutral run envelope with millisecond lifecycle timestamps
• backend_run_snapshot_events — ordered snapshot event stream for stable harness phases such as model text/reasoning boundaries, tool input/output availability, usage, and finalization
• GET /observability/runtime-snapshot — live server/runtime resource view that includes server process health, active chat runs, replay buffers, provider runtime hosts, PTY resources, Chronicle daemon resources, latest desktop samples, renderer/browser drill-downs, and observability queue health
• OpenTelemetry metrics — low-cardinality gauges/counters derived from runtime samples; use only as trend/correlation evidence during debugging
• Server logs — pino-structured JSON written to {CRADLE_DATA_DIR}/server.log (or $CRADLE_LOG_FILE)

Observability and run snapshot rows are forensic records. Session/run/message foreign keys may be NULL after source rows are deleted; do not treat a null FK as malformed data. Snapshot retention is controlled by CRADLE_CHAT_RUN_SNAPSHOT_RETENTION_DAYS (default 30, 0 disables pruning).

Keep these snapshot types separate:

• Durable run snapshots (backend_run_snapshots, backend_run_snapshot_events) answer "what happened during this chat run?"
• Runtime snapshots (/observability/runtime-snapshot) answer "what resources and live runtime state exist right now?"
• Metrics answer "does the low-cardinality trend agree with the live/runtime evidence?"

DB path resolution

The script tries in order:

1. --db <path>
2. $CRADLE_DB_PATH
3. $CRADLE_DATA_DIR/cradle.db
4. ~/Library/Application Support/Cradle/cradle.db (macOS)
5. ~/.config/Cradle/cradle.db (Linux)

Log path resolution

The logs command tries:

1. --log <path>
2. $CRADLE_LOG_FILE
3. $CRADLE_DATA_DIR/server.log
4. <db-dir>/server.log (sibling of the resolved cradle.db)

Script

Use:

python3 resources/skills/cradle-observability-debugger/scripts/obs_debug.py --help

Use the generated Cradle CLI first when the server is running; it exercises the same HTTP API used by agents and preserves server-side redaction/export behavior:

pnpm --filter @cradle/cli cradle observability --help pnpm --filter @cradle/cli cradle chat snapshot --help

Use the SQLite script when the server is down, the HTTP API is suspect, or you need to inspect local DB state without starting Cradle.

Commands

summary — snapshot health

python3 resources/skills/cradle-observability-debugger/scripts/obs_debug.py summary --since-min 120

Prints counts by code/severity and top open incidents. There is no generated CLI equivalent for this aggregate summary.

events — query observability events

pnpm --filter @cradle/cli cradle observability events --code CHAT_EMPTY_OUTPUT_COMPLETION --limit 50 pnpm --filter @cradle/cli cradle observability events --chat-session-id <id> --limit 200

python3 resources/skills/cradle-observability-debugger/scripts/obs_debug.py events --code CHAT_EMPTY_OUTPUT_COMPLETION --limit 50 python3 resources/skills/cradle-observability-debugger/scripts/obs_debug.py events --chat-session-id <id> --limit 200

incidents — query incidents

pnpm --filter @cradle/cli cradle observability incidents --status open --limit 50 pnpm --filter @cradle/cli cradle observability incidents --code TURN_STREAM_FAILED --limit 200

python3 resources/skills/cradle-observability-debugger/scripts/obs_debug.py incidents --status open --limit 50 python3 resources/skills/cradle-observability-debugger/scripts/obs_debug.py incidents --chat-session-id <id> --limit 200

error-patterns — query grouped failure signatures

pnpm --filter @cradle/cli cradle observability error-patterns --limit 50 pnpm --filter @cradle/cli cradle observability error-patterns --run-id <runId>

timeline — run snapshot history

pnpm --filter @cradle/cli cradle chat snapshot run <runId> pnpm --filter @cradle/cli cradle chat snapshot session <sessionId>

python3 resources/skills/cradle-observability-debugger/scripts/obs_debug.py timeline --run-id <runId> --limit 500 python3 resources/skills/cradle-observability-debugger/scripts/obs_debug.py timeline --chat-session-id <id> --since-min 240

Timeline output is an array of backend_run_snapshots; each item includes parsed summary_json plus ordered events from backend_run_snapshot_events. Snapshot/event timestamps are milliseconds.

runtime-snapshot — live runtime resources

pnpm --filter @cradle/cli cradle observability runtime-snapshot --json pnpm --filter @cradle/cli cradle observability runtime-snapshot --json server,chatRuntime,providerRuntime,pty,observability

Use this when debugging leaks, stuck runs, replay buffer growth, lingering provider hosts, PTY descendants, Chronicle daemon resources, desktop renderer memory, or observability queue backpressure.

Important fields:

• server.memory, server.cpu, server.node — process RSS/heap, CPU window, active handles, and active requests
• chatRuntime.activeRuns — live run IDs, sessions, provider target kind/id, and model ID
• chatRuntime.replayBuffers — buffered chunk and delta counts per active run
• providerRuntime.hosts — runtime host ref counts, pin counts, resource ownership, and expiry
• pty — terminal resources, process descendants, RSS, and CPU by role
• chronicle — Chronicle daemon resource state
• desktop.latestSamples — raw Electron main/app metrics, windows, BrowserPanel diagnostics, and renderer diagnostics reported by desktop main
• drilldowns.renderer — top renderer windows, top chat sessions by estimated retained chars, and active streaming messages; use after Grafana shows renderer heap or chat payload pressure
• drilldowns.browserPanel — BrowserPanel owner/thread/tab/runtime to WebContents/process mappings; use when Tab working set grows without matching renderer JS heap growth
• drilldowns.replay.topRuns — active runs with the largest replay buffers, joined back to session/message/provider/model IDs
• drilldowns.providerRuntime.topHosts — provider hosts sorted by ref/pin/resource/idle pressure, with expiry and idle timing
• observability — event queue depth/drop/persistence health

This command is server-backed only; it has no SQLite fallback because it reports live process state.

Runtime snapshot drill-down sequence for memory and retention issues:

1. If drilldowns.renderer.topChatSessions is large and aligns with renderer heap growth, inspect that session's durable snapshots/messages.
2. If drilldowns.renderer.rendererWindows shows high Tab/renderer identity but chat sessions are small, inspect drilldowns.browserPanel.liveTabs and drilldowns.browserPanel.runtimes for retained WebContents/native BrowserPanel state.
3. If stream/replay metrics stay non-zero, inspect drilldowns.replay.topRuns and drilldowns.renderer.activeStreamingMessages before checking logs.
4. If provider host metrics stay non-zero after active runs fall, inspect drilldowns.providerRuntime.topHosts for ref count, pin count, expiry, and idle duration.
5. If metrics disagree with drill-downs, suspect sampler/exporter lag or stale Prometheus range data before blaming the resource owner.

metrics — trend and exporter cross-check

Use metrics as secondary evidence after runtime-snapshot, not as the first source of truth. The useful debug question is whether the exported low-cardinality trend agrees with the live JSON snapshot and recent incidents/logs.

Debug sequence:

1. Capture cradle observability runtime-snapshot --json and identify the suspect resource family: server memory/handles, active runs, replay buffers, provider hosts, PTY resources, Chronicle resources, desktop samples, renderer/browser drill-downs, or observability queue health.
2. Check the metric backend for the matching cradle_* series over the same time window.
3. If JSON shows growth but metrics are flat or missing, suspect sampler/exporter/configuration drift rather than the resource owner.
4. If both JSON and metrics grow, continue with owner-specific evidence: drilldowns.renderer.topChatSessions, drilldowns.browserPanel.liveTabs, active run IDs, replay top runs, provider host IDs, PTY descendants, queue depth, or logs.
5. If metrics grow but JSON is currently clean, treat it as a historical spike and correlate with incidents, durable run snapshots, or server logs around the metric peak.

Do not spend time teaching telemetry setup from this skill. If metrics are unavailable, continue with runtime-snapshot, SQLite events/incidents/timeline, and logs.

logs — server log

python3 resources/skills/cradle-observability-debugger/scripts/obs_debug.py logs --tail --lines 50 python3 resources/skills/cradle-observability-debugger/scripts/obs_debug.py logs --filter "mapper" --lines 200 python3 resources/skills/cradle-observability-debugger/scripts/obs_debug.py logs --filter "error" --tail --lines 30

Logs are pino-structured JSON. Pipe through python3 -m json.tool to pretty-print individual lines.

bundle — deterministic export

pnpm --filter @cradle/cli cradle observability export \ --chat-session-id <chatSessionId> \ --since-unix <unixSeconds>

python3 resources/skills/cradle-observability-debugger/scripts/obs_debug.py bundle \ --chat-session-id <chatSessionId> \ --since-min 240 \ --out /tmp/cradle-obs-bundle.json

The bundle includes observability events, incidents, and run snapshot timelines with metadata.

Typical debug flow

1. If the server is running, start with cradle observability error-patterns --limit 50 or cradle observability incidents --status open.
2. Use cradle observability events --code <CODE> --limit 50 to narrow to one session/run.
3. Use cradle chat snapshot run <runId> or cradle chat snapshot session <sessionId> to inspect durable harness phases.
4. Use cradle observability runtime-snapshot --json when the symptom looks live-resource related: memory growth, stuck active runs, retained replay buffers, provider host leaks, PTY process leaks, Chronicle daemons, desktop renderer pressure, BrowserPanel/WebContents retention, or observability queue backpressure.
5. Use metrics only as a trend/exporter cross-check; compare them with runtime-snapshot before blaming the resource owner.
6. Use logs --filter "<chatSessionId>" --lines 100 when DB/API evidence is not enough.
7. Use cradle observability export ... for API-faithful sharing; use the script bundle when the server is unavailable.

Guardrails

• Do not mutate DB state from this skill.
• Prefer filtering by chatSessionId or runId before broad scans, but remember old forensic rows may have null FKs.
• If neither chatSessionId nor runId is known, run summary then events --code ... first.