adrianco/retort/diagnose-failed-run

Diagnose a Failed Retort Run

Overview

A run failing the mechanical gate (test_coverage=0 → all metrics zeroed → status=failed) says only that the tests didn't run/pass under the scorer — not that the model produced bad code. Historically most "failures" in this repo were harness/measurement artifacts, not model defects. This skill is the judgment layer on top of retort diagnose: it establishes the real cause with direct evidence before anyone attributes, reports, or gates on it.

It exists because we repeatedly drew wrong conclusions by inferring from a signature ("flaky", "concurrency", a wrong env var, a wrong tool-permission fix) that end-to-end checks later overturned. The cost of guessing here is hours and bad decisions.

Cardinal rules

1. Ground-truth before concluding. Run the tests yourself, inspect the workspace, read

the agent logs. Never attribute a cause from the gate verdict, a duration, or a single log line alone.

1. **Label every claim [DIRECT] (observed: command output, file state, a reproduced

result) vs [HYPOTHESIS] (inferred, not yet tested).** Don't ship a hypothesis as a cause. If a cause is unconfirmed, say so and name the experiment that would confirm it.

1. Read the DB *and* the agent logs. The recorded scores/session show which tool or

step failed; the agent stderr shows why (the permission type, the error). You usually need both — e.g. the session said "read rejected", but only --print-logs stderr revealed the real permission was external_directory.

1. Verify any fix end-to-end (through the harness), not just CLI/unit. A fix that

passes a CLI probe can still fail in the runner (a wrong env var passed both).

Inputs in each runs/<cell>/repN/

If _agent_*.log are absent, the run predates log capture; reconstruct from the agent's own session db (e.g. opencode's OPENCODE_DB sibling) and add capture before re-running.

Procedure

1. Mechanical pass first: retort diagnose --experiment-dir <dir> labels each failed

run TOOLING (recovers with retort rescore --only-failed) vs GENUINE. Start here; this skill handles the cases it can't auto-resolve.

1. Ground-truth each failure (do NOT skip even for GENUINE):

• Does code exist, and where? A "no code" verdict is often just a package subdir

(app/, src/) the eye/glob missed.

• Run the tests yourself in a clean copy with deps installed

(pytest / cargo test / go test ./... / mvn test / …). Passing tests ⇒ it's a scorer false-fail, not a model miss.

• Read `_agent_stdout.log` for the failing tool call + status; **read

_agent_stderr.log** for the reason.

1. Classify the cause (see taxonomy). State it [DIRECT] with the evidence.
2. Isolate intermittent/ambiguous causes with a controlled A/B — change exactly one

variable (e.g. shared vs isolated db; baseline vs with-fix batch) and compare. Log what you dropped; don't conclude from n=1.

1. Reproduce-with-full-capture before declaring an intermittent cause — run it enough

times to catch one with _agent_*.log captured, then read the events at the abort.

1. Verify the fix end-to-end through the harness, then re-score / re-run.

Cause taxonomy

A. Infrastructure false-fail (code works; the harness mis-scored it → fix the harness, then rescore):

• Scorer gap — language/runner the scorer doesn't handle (e.g. a Bun bun:test project the TS scorer can't measure).
• Undeclared / transitive deps — model omits requirements.txt; tests fail at import (incl. transitive deps like httpx for fastapi/starlette TestClient).
• Permission auto-deny — headless agent denies a tool (opencode's external_directory: ask→deny on the temp workspace) → aborts with no code.
• Concurrency/env contention — shared state (opencode's single opencode.db) → fast-fail at startup under parallelism.
• Mis-measurement — coverage parsed as 0 on passing tests, etc.

B. Genuine model miss (real, count it; not fixable in the harness):

• Won't compile / tests genuinely fail / no tests written / no or partial deliverable / code in the wrong place the task didn't ask for.

C. Environment (transient/operational, not the model or harness logic):

• Rate-limit / credit cutoff (check the provider key's usage/limit), stray processes, resource exhaustion, OS cleanup.

Tells (and their trap)

The classic tell — fails instantly for ~$0 ⇒ harness; burns model time ⇒ genuine — is a starting hint, not a verdict. It breaks when scoring is the failure point: a run can burn minutes producing working code and still be scored 0 (scorer false-fail). And a fast no-code abort can be a harness permission denial, not a model giving up. Always ground-truth.

Output

Per failure, report: cell · cause (one of the taxonomy) · [DIRECT]/[HYPOTHESIS] with the evidence · recommended action (retort rescore; fix scorer/harness then rescore; count as genuine; re-run with capture). Then the rolled-up verdict: how many failures are infra vs genuine, and whether the infrastructure is trustworthy enough to proceed.

Logging setup (prerequisite)

• The runner persists _agent_stdout.log / _agent_stderr.log per run (_persist_agent_output).
• For opencode, the harness passes --print-logs so stderr carries the diagnostic logs

(permission evaluations, step loop). Raising opencode's level (--log-level DEBUG, OPENCODE_LOG_LEVEL=verbose) adds ~nothing — the --format json stdout stream is the signal.

See also

• retort diagnose / retort rescore — automated TOOLING/GENUINE triage + recovery.
• evaluate-run skill — the per-run spec/quality evaluation (this skill is for failures).