radiantlogicinc/fastworkflow/debug-workflow-conversations

Debugging workflows from conversation logs

Use recorded turns and their span trees to identify the first wrong decision and its consequences. Capture depends on redaction, limits and writer health; not every run contains every input or response.

1. Locate the conversation and its evidence store

In an existing run_chatbot session, navigate benchmark → experiment → conversation, or ad-hoc conversations → UTC date → conversation. The left tree stops at conversations. Select turns and drill into phases, steps and calls on the right; the right breadcrumb shows the complete path. Read the Human feedback history at the relevant component as well as its raw evidence. Use optimize-workflow-with-feedback when the task is to improve and compare the workflow, rather than diagnose one failure.

For ordinary runs, resolve the database through the framework:

from fastworkflow import state_paths
db_path = state_paths.observability_db("<workflow_folder>")

For registered experiments, resolve the registration's store via benchmark_setup.load_experiment(workflow_folderpath, experiment_id). A null store means execution has not bound the registration yet. Retain store_id with every turn/span reference. Conversation IDs are local to channels/stores.

A missing default database does not prove the workflow never ran: check the selected state root and experiment registration first. If reproduction is needed, prepare it under the project's execution permissions. The supported picker command is fastworkflow run_chatbot; selecting a workflow can start a server, so it is not merely an offline file viewer.

2. Read it — read-only, always

from fastworkflow.observability.store import ReadOnlyObservabilityStore
store = ReadOnlyObservabilityStore(db_path)

Never instantiate `ObservabilityStore` to inspect a database. That class is the writer: constructing it can create the file and write-probe it, and it DELETES a populated database from an older schema version and recreates it empty (there is no migration). A database from a newer version is refused. Preserve older evidence by copying it first, and read it with the matching framework version rather than altering its schema. ReadOnlyObservabilityStore opens mode=ro connections and cannot mutate anything. Raw SQL is equally fine (the schema is documented in reference.md):

sqlite3 "file:$DB?mode=ro" "SELECT turn_key, status, success, user_message FROM turns ORDER BY turn_key DESC LIMIT 20"

3. Find the failing turn

store.list_turns(status="failed")                 # turn-level failures
store.list_turns(success=False)                   # any command in the turn failed
store.list_turns(command_name="cancel_order")     # turns that executed a command
store.list_turns(context="TodoList")              # substring match on entry context
turn  = store.get_turn(turn_key)                  # full row incl. record_json
spans = store.get_spans(turn_key)                 # the trace, ordered by start_ns
comments = store.list_human_feedback(turn_key)    # all human annotation anchors for this turn

Two orthogonal outcome fields, both worth reading:

• status is the turn lifecycle: completed / failed (agent ran out of

iterations — failure_reason: max_iters_exhausted) / awaiting_user (still suspended) / cancelled / abandoned.

• success means every command in the turn succeeded. **completed with

success = 0 is the case to hunt**: a command failed and the agent wrote a confident answer over it.

4. Walk the trace to the failing stage

A turn's spans nest like this (parent_span_id links them; names and full attribute catalogs in reference.md):

fw.turn                          the whole logical turn (+ context_mutations diff)
├── fw.planner.plan/.replan      the agent's plan; replans carry their trigger
│   └── fw.llm.call              the planner's LM call
├── fw.skill.phase               one phase of a phased skill call (one per phase run;
│   └── fw.agent.execute         its budget, steps used, facts filled or facts_error)
└── fw.agent.execute             the ReAct loop as a whole (attempts, final answer)
    └── fw.agent.step            one reasoning step: thought, tool choice, observation
        ├── fw.llm.call          the step's reasoning LM call (cache_hit exposes
        │                        stale-cached completions)
        └── fw.agent.tool_call   the agent invoking a command (raw_command)
            └── fw.command.execute       resolution + execution
                ├── fw.nlu.intent            matcher layer, confidence, candidates
                └── fw.nlu.param_extraction  extraction + validation, structured
                    └── fw.llm.call          the LM extraction call, when one ran
fw.ask_user                      a clarifying question + the human wait
 
Deterministic "/"-mode turns skip the planner/agent layers (fw.command.execute
directly under fw.turn); assistant-mode tool_call/command.execute pairs sit
under fw.turn without agent.step.

5. The triage tree

Inspect the recorded causal order. The checks below help locate the first wrong decision; a bad plan can precede routing, and a provider or backend failure may be environmental. Treat a human comment as a lead to verify, not proof of the cause.

A. Did routing pick the right command? Compare fw.command.execute's raw_command (what was asked) against its command_name (what ran), then read the fw.nlu.intent spans:

B. Were the parameters extracted correctly? Read fw.nlu.param_extraction:

C. Did the agent plan a workable sequence? fw.planner.replan spans carry their trigger: repeated parameter_extraction_error replans mean the agent cannot discover where a handle comes from → declare-parameter-producers. Repeated ask_user_response replans point at ambiguous command surfaces or missing context navigation → design-context-models.

D. Did the conversation stall on questions? Multiple fw.ask_user spans in one turn (each records agent_query and the reply): review whether each question was needed. Separate repeated asks after a valid answer from necessary clarification or retries after invalid answers; inspect state retention as well as B.

E. Was state stored and used? The fw.turn close carries context_mutations (added / changed / removed keys with brief values). A command that should have stored a handle but shows no mutation, or a later turn that re-asks for stored information, is a storing-information-in-context bug in the command's own code.

F. Everything above clean? Then the failure is the command implementation: fw.command.execute has status: error or success: false with the response_text; the full CommandResponse (and, when FW_OBS_CAPTURE_TRACEBACKS=1 was set, the traceback artifact) is in the turn's record_json. Read the command's _commands/<name>.py source next.

6. Recommending fixes

State the diagnosis with its evidence (span attribute values, not paraphrase), name the feature, and load the companion skill before writing the change:

Human feedback and output-quality failures

Read reference.md for annotation anchors and read APIs. human_feedback comments are distinct from agent-memory feedback and formal review ratings. Do not write an annotation merely to inspect a trace or present an agent judgment as a human one.