darwin-agent/harnessx/analyze

Analyze — reading trajectories, extracting experience

Your input is a directory of <task_id>.md files, one per task in this round. No pre-computed summaries, no cluster files — you investigate the markdown directly. Your output is a set of candidate hypotheses, each naming a concrete pattern (either a gap to close or a habit to preserve / generalize) and the lever you'd pull.

Passing and failing trajectories are treated as equal evidence sources. A passing cluster can teach you a habit worth locking in or worth lifting into the template for the failing cluster; a failing cluster can teach you a blocker worth closing. Both feed the same candidate pipeline — same template, same retroactive check, same global-first ordering. The bias to read only failures is the single biggest cause of "right diagnosis, still-net- negative round": fixes land but previously-working habits silently regress because no candidate preserved them.

Trajectory file shape

YAML frontmatter + markdown body. Read <path> limit=30 returns the frontmatter plus the first few body lines — usually enough to classify the task. When the frontmatter hints at a deeper cause, read the last ~40 lines of the body to see what the agent actually had in context at its decisive step.

Frontmatter tiers (render order)

Behaviour — always present

• exit_reason — done / max_steps / budget_exceeded /

loop_detected / error

• steps, total_tokens, cost_usd, elapsed_s
• tool_call_counts — {tool_name: n}
• tool_error_counts — {tool_name: n}
• error_count, pivotal_tool
• final_output_length

Eval — always present, authoritative

• eval_passed (bool) — the external pipeline evaluator's verdict
• eval_score (float) — numeric score (0.0 / 1.0 for exact-match)

eval_* is the ground-truth correctness signal. When Eval and Judge disagree, believe Eval. The dataset's expected answer text is intentionally withheld from the frontmatter, and so is the evaluator's textual reason — you see only pass/fail and score, never the target string. Evolve on correctness signals, not on a known target. For the why behind a failure, read the trajectory body and the optional judge_* fields below.

Judge — optional

Present only when LLMJudgeProcessor ran. An LLM's opinion, not ground truth. Useful for why a task failed, not whether.

• judge_verdict — plausible / unsupported / hedging /

format_wrong / refused / no_answer / judge_error

• judge_cause, judge_missing, judge_lesson — one-line strings
• judge_missing_capability — {present: bool, summary: "..."};

primary signal for Action-lever decisions

• extracted_answer — what the agent actually committed (the

expected answer text is never exposed; do not try to reconstruct it)

judge_error is the judge failing (timeout, bad JSON), not the agent. Treat as "no opinion" and lean on behaviour + body.

The two axes of reflection

Every candidate hypothesis is pinned on two axes. Keep them independent — don't collapse the lens into the lever.

Axis A — Lens (what is the pattern?)

Three reading passes over the same batch, not three buckets. A single trajectory can answer more than one question — you are sweeping for signals, not assigning labels. Every pass anchors on the authoritative eval_passed field; behaviour / judge signals describe the shape, never the verdict.

exit_reason=done alone is not evidence of success — the agent merely called end_turn. Always gate on eval_passed before reading a trajectory as a positive example.

The three lenses feed the same candidate pipeline. A Success- lens observation turns into a candidate in one of two shapes:

• Lock-in — encode the habit as Configuration / Control /

Instruction so a future round's edit cannot silently regress it. Justification: the cited passing tasks would fail if the habit were removed.

• Transfer — lift the habit into a rule applied to a failing

cluster. Justification: the cited failing tasks would pass if the habit were applied.

Failure and Capability-gap lenses produce corrective candidates (close a gap). Both polarities go through the same retroactive check, same global-first ordering, same candidates.md schema. What changes is which variant of the retroactive check the candidate has to answer.

Rushing to Failure and skipping the other two leaves Capability-gap bets (blocked-but-sometimes-recoverable shapes) and Success-pattern lessons (what to preserve or generalise) on the table.

Axis B — Lever (where does the fix belong?)

The axes are orthogonal. A Failure symptom can be fixed at any of the four levers. A Capability-gap lens does not always mean a new tool — if the existing tool fires but returns unusable data, the lever is often Control (an on_after_tool processor that parses), not Action.

Worked examples — lens × lever × intent

The three axes cover 3 × 4 × 3 combinations; most are rare. Below are representative shapes you'll actually see. Use them to pressure-test your own candidate — if yours doesn't resemble any row here, either you've found something genuinely new or the lens / lever / intent tag is wrong.

Rows omitted on purpose: Capability-gap / Instruction is almost always wrong (if the capability isn't there, no prompt rule conjures it — covered in "Action vs Instruction" below); Success / Configuration is rare because knob values don't encode a habit; Success / Action is rare because authoring a new tool is not a natural way to preserve a habit the existing tools already enable.

If your candidate lands in an omitted cell, that's a signal to challenge the lens / lever / intent tag before the retroactive check.

Read strategy

Your output is labelled clusters with cited evidence, not a fixed procedure run. The trajectories live under _meta_scratch/traj/ as <task_id>.md files. Available tools: Read / Grep for direct inspection, spawn_reflect_worker(kind="trajectory- digester", files=[...]) for batching when the task count would blow the token budget from TASK.md. Pacing — how many frontmatters to scan before clustering, how many representatives to body-read per cluster, when delegation is worth it — is your judgement call. Record the reasoning in candidates.md.

Two constraints are non-negotiable:

• Systemic vs idiosyncratic. A cluster (failing OR passing)

earns a config change only when the same shape recurs across at least two distinct tasks with different inputs. One-off observations — however obvious the fix or however clean the habit feels — go to NEEDS_FROM_HUMAN and are skipped. Two tasks that look different but share the same root cause still count as one "gap" (test: if you fix one, does the other follow?); two tasks with coincidentally similar frontmatter but different mechanisms do not count as a cluster.

• **Frontmatter shows the ceiling shape; the body shows what

actually happened at the decisive step.** Don't propose Action or Control on frontmatter signals alone — for failure candidates a body must name the actual failure mode (raw bytes, empty results, redirect HTML, cycling URL, silent parse failure); for success candidates a body must show the habit actually firing (the ## Plan block, the retry loop, the tool sequence) rather than just inferring it from short step count. A candidate whose evidence cites only a worker digest, with no trajectory body quoted, is not grounded.

The retroactive check — three variants, one per candidate intent

Before writing any code, for each candidate, ask the variant that matches the candidate's intent. A candidate has exactly one intent; one that seems to need more than one variant is two candidates glued together — split them.

Variant A — Corrective (Failure / Capability-gap lens)

If this fix had already been in place on the cited failing

trajectories, would the task have succeeded?

• A commit-nudge requires an answer to commit.
• A template rule requires data to apply.
• A tighter loop detector requires the loop to be the actual

blocker, not the consequence of the real blocker.

• A new tool requires the task to have needed that tool's output

shape, not just the tool's topic.

When the answer is "no — the agent would still have failed", the symptom is downstream of an earlier failure. Trace one step back: what did the agent lack, not what did it fail to do? That upstream gap is the real lever. Drop or re-scope.

Variant B — Preservative, lock-in (Success lens)