scottrbk/eval-harness/eval-interpretation

Overview

This guide covers how to monitor an eval-harness run while it is in progress, interpret the logs and results once it finishes, and diagnose the failures that can occur along the way.

Locate the run

Every run writes to its own directory: <OUTPUT_DIR>/<YYYYMMDD_HHMMSS>_<session_id>/, where OUTPUT_DIR is set by EVAL_HARNESS_OUTPUT_DIR (default output). Inside it:

• session.log - every record from every agent thread, interleaved.
• <agent_type>_<model>[_<effort>][_id].log - one per agent, containing only that agent's

records. Variant IDs and profile names distinguish baseline/treatment runs; characters that cannot appear in a filename (e.g. / in model names) are replaced with _.

• results.json (or results.csv) - written when the session finishes; see the

Results File Schema for the field-by-field breakdown.

• diagnostics/<agent>/eval-XX/run-XX/attempt-XX/ - present only for failed attempts when

EVAL_HARNESS_CAPTURE_FAILURE_DIAGNOSTICS=true; contains failure metadata and best-effort AgentShell trace/container state.

Read the logs

Log lines are formatted <timestamp> - <level> - <file:line> - <message>. The markers to look for:

• --- capabilities phase --- / --- arrange phase --- / --- act phase --- /

--- score phase --- - phase boundaries. The capabilities phase is present only for a non-empty profile and runs before arrange.

• [arrange] <line> (likewise [act] and [score]) - stdout/stderr streamed from inside the

container, including every print() in the eval's phase methods.

• Run 1/3 - logged when an eval has run_count > 1; each run is a fresh container.
• EVAL_TOTAL_TOKENS=<int> - emitted once per agent-shell call and summed into the token totals.
• EVAL_SCORE=<float> - printed by the eval's score method; the harness takes the last

occurrence in the score phase output. If no line matches, the run silently scores 0.0.

• phase <label> completed - the phase finished cleanly.

In the results file, an eval's score is the mean across its runs, while total_tokens and time_taken_seconds are totals across them.

Monitor a live run

tail -f the session log, or a single agent's log for less noise. A failing agent does not stop the others: it is logged as Agent <type>-<model> failed with a traceback, marked FAILED, and the session carries on. At the end, main.py prints one FAILED: ... line per failed agent and exits with code 1.

Diagnose failures

Where the fix belongs

Classify the failure before changing anything:

• The eval itself (authoring constraint violations, scoring bugs) - fix the eval class,

following the eval creation skill.

• Harness configuration (timeouts, output locations) - .env / EVAL_HARNESS_ environment

variables, documented in Configuration.

• Credentials - re-authenticate on the host per the

authorisation guide.

• The docker image - rebuild manually after any Dockerfile or agent-shell change; there is no

auto-rebuild.