darwin-agent/harnessx/gaia-playbook

GAIA: Benchmark-specific Guidance

Task Structure

Per the GAIA paper's own frequency table, in decreasing order of how often each capability is needed:

1. Web browsing — visit a specific page, not just read a search

snippet

1. Code execution — compute from retrieved data (pandas, math)
2. File handling — parse .pdf, .xlsx, .csv, .docx,

.mp3, .mp4, .jpg, .zip

1. Multi-modal understanding — images, audio, video, PDFs with

figures

1. Long-horizon planning — Level 3 tasks can require 20-50 steps

Common question shapes: "According to <source>, what is <value>?", "How many X in Y between Z1-Z2?", riddles, reversed/encoded text, embedded-file computations, YouTube video questions.

Analyzing Failures

Primary-evidence frontmatter fields (always reliable):

The dataset's expected answer and the evaluator's textual reason are both intentionally withheld from the frontmatter — the meta-agent sees only pass/fail and the numeric score. For the why behind a failure, read the trajectory body and the optional judge_* fields below.

Opinion-only frontmatter fields (present only when the LLM judge processor ran):

When eval_passed and judge_verdict disagree, trust eval_* for pass/fail accounting and use judge_* to probe why. judge_error is the judge itself failing (timeout, bad JSON), not the agent — do not build a candidate off those rows.

(Generic trajectory-reading methodology — lens × lever × intent, three-variant retroactive check, systemic-vs-idiosyncratic filter, candidate schema — all live in the analyze skill. This playbook only layers GAIA-specific patterns on top. In particular the "would it pass with unlimited steps + perfect discipline?" question is just Variant-A retroactive check with GAIA framing.)

When diagnosis points at a missing capability

GAIA is hard on purpose: many failing tasks would still fail with more steps, a better prompt, or a tighter loop guard, because the agent literally does not have the capability the task requires. These are your highest-leverage opportunities — answer the second question above honestly ("with unlimited steps and perfect discipline, would the final state still lack the answer?") and when the answer is yes, the default response is to author a new tool, not to nudge the model.

Signals that you are in a capability gap (not a behaviour gap):

• judge_missing_capability.present = true with a non-empty

summary — the judge is naming the shape of the missing tool for you. Use summary as a starting sketch, not as a spec.

• judge_missing is non-empty and repeats across a cluster of

failing tasks (e.g. "PDF table parser" on 3+ rows, "JS-rendered page browser" on 4+ rows). Repetition is evidence the gap is a class, not a one-off.

• Failure modes E (raw bytes), F (missing modality), G (Wayback /

non-HTML endpoints), and most H (oversized context) post-mortems resolve to new tools rather than processors. When you see these, go straight to Action-layer candidates.

• The trajectory shows many retries against the same endpoint, or

the agent paraphrases instead of quoting — both are "tool returned nothing usable", not "agent chose badly".

How to act on it:

1. Read _meta_scratch/CONTEXT.md when present — it's the machine-

rendered lever scoreboard from the journal, so you can see at a glance whether instruction/control has already been tried on each task without lift. Use its hypothesis history as the seed for steps 2-4 below.

1. Read reference skill in the meta-harness workspace — it has the

authoring mechanics (shape spec in TOOL_SPEC.md before code, stdlib-first impl, config wiring, end-of-turn checklist). The replay gate that runs after end_turn is the integration oracle — no self-written probe.

1. Consult the "GAIA capability classes" table later in this playbook

— match your cluster's judge_missing_capability summaries to a row, use its "Shape hint" as a framing for your TOOL_SPEC.md. Do not copy the hint verbatim; your inputs differ from past teams'.

1. Follow the recipe's checklist. Re-run the cluster on the new tool

and check the gap closed. If judge_missing_capability.present still fires on those rows, the tool shape was wrong — revise before declaring the candidate.

Conversely, do not reach for a new tool when:

• judge_missing_capability.present = false — judge already said

the problem is behavioural.

• The answer is present in the trajectory body but the `FINAL

ANSWER:` marker / format is wrong (mode C). That is a Control layer fix.

• A single existing tool already covers the capability and only the

instruction to use it is missing. That is an Instruction lever.

Authoring a tool to dodge a prompt fix inflates the action surface without lifting the score. Authoring a prompt fix to dodge an obvious capability gap keeps the benchmark locked at the ceiling of what the current tools can reach.

Known Techniques That Improve GAIA Scores

These are generic GAIA wisdom from public writeups by teams scoring 40-75% on the leaderboard. Each is mapped to the lever you'd use in HarnessX.

Don't just map failures to existing tools. The table is a menu of techniques with known lifts — but GAIA's ceiling on any one scaffold is set by the tools the scaffold has. Teams that broke through 60%+ did it by writing new tools that did not exist before (HuggingFace's markdown browser, their per-extension file inspector). When a cluster of failures points at a missing capability your current Action list does not cover, the right move is to author a new tool sized to that cluster — see ### When diagnosis points at a missing capability above. Pattern- matching a hard task to the closest-fitting existing tool is how benchmarks stall.

GAIA capability classes that have unblocked past systems

When the journal flags a cluster and you write _meta_scratch/TOOL_SPEC.md (per the reference skill's "Before you type — write the shape spec" section), it helps to know which tool classes have historically unblocked top teams on GAIA. This is vocabulary, not a catalogue to copy from.

The pattern is: match your cluster's judge_missing_capability summaries to one of the classes below, then write a SHAPE from YOUR failing trajectory inputs. Do not copy implementations from elsewhere — those were shaped for different input distributions than this run's.