adrianco/retort/compare-runs

Compare Runs

Overview

Retort is a Design of Experiments engine — every run sits at a point in the factor space, and replicates exist to measure noise. Unlike pourpoise's compare-attempts (which maintains a ranked leaderboard of ad-hoc submissions), comparing retort runs is about factor effects and qualitative divergence, not "who won".

This skill is the qualitative counterpart to retort analyze / retort report effects. The ANOVA tells you whether a factor is significant. This skill tells you what changed in the generated code as you varied it.

Parameters

• experiment_dir (required): e.g. experiment-1/. Must contain retort.db and a runs/ directory of per-run archives, each already evaluated by evaluate-run (i.e. each rep<N>/ has an evaluation.md and findings.jsonl).
• output_file (optional, default: {experiment_dir}/reports/comparison.md): Where to write the comparison report.
• group_by (optional, default: all factors): Comma-separated factors whose effect you want to isolate. The other factors are aggregated over.
• include_failed (optional, default: true): Whether to include failed runs in the comparison (they're still informative — failure itself is a finding).

Steps

1. Discover evaluated runs

find {experiment_dir}/runs -name evaluation.md -not -path "*/salvaged-*/*" | sort

Parse each path to extract the cell (from the parent directory name, e.g. language=rust_model=opus_tooling=beads) and replicate (from the rep<N> or rep<N>-failed segment).

Constraints:

• You MUST skip runs under salvaged-*/ — those are manually-archived artifacts from before auto-archival existed, and their layout is not regular.
• You MUST handle both rep<N>/ (success) and rep<N>-failed/ (failure) directory names.
• You SHOULD warn but not fail if some cells have no evaluation (evaluator hadn't run yet).

2. Load findings and metrics

For each evaluated run, load:

• Factors (from the cell directory name or the run's stack.json)
• Summary metrics (from evaluation.md header: requirements pass count, test counts, build status)
• Findings (from findings.jsonl) — categorize by kind and severity
• Architecture signal (from summary/index.md: the "Shape" one-liner)

Constraints:

• You MUST NOT re-evaluate. If evaluation.md is stale relative to the source, add a stale tag but use what's there.
• You MUST preserve replicates separately — do not average them into the cell yet.

3. Aggregate per cell

Group runs by cell (same factor combination across replicates):

Constraints:

• You MUST show mean ± sample standard deviation, not just mean. Variance across replicates is a first-class signal.
• "Distinct shapes" counts unique architecture summaries across replicates — a cell where the agent chose a different framework each time is a signal worth surfacing.
• Cells with only one replicate: drop the ±sd and mark as n=1.

4. Surface factor effects

For each factor in group_by, hold the others roughly fixed and report the effect:

## Effect of `tooling` (none vs beads)
 
Aggregating over language and model:
 
| Tooling | Mean req pass | Mean token cost | Mean build time | Build fail rate |
|---------|---------------|-----------------|-----------------|-----------------|
| none    | 8.1 / 10      | 289K tokens     | 124s            | 1/18 (6%)       |
| beads   | 7.2 / 10      | 441K tokens     | 156s            | 2/18 (11%)      |
 
Direction: beads costs ~52% more tokens and scored 11% lower on requirements.
This matches the p < 0.10 effect reported by `retort analyze`.
 
Qualitative: with `tooling: beads`, agents spent ~30% of their turns
on bd bookkeeping (visible in run transcripts). This accounts for the
token overhead.

Constraints:

• You MUST NOT claim statistical significance yourself — cite retort analyze or note "directional only".
• You MUST cross-reference the structured effect size reported by retort report effects when available at {experiment_dir}/reports/analysis/*.md.
• Qualitative claims MUST cite specific run IDs or findings (see rep3 of lang=python,model=opus,tooling=beads).

5. Identify qualitative divergence

Find cells where runs within a replicate group diverged in a way pure metrics miss:

## Qualitative divergence
 
### lang=typescript, model=sonnet, tooling=beads — 3 replicates, 3 different shapes
 
| Rep | Framework | Storage | Notes |
|-----|-----------|---------|-------|
| 1 | Express + better-sqlite3 | SQLite | Built, tests failed on migration |
| 2 | Fastify + Prisma | Postgres (!) | Didn't build — assumed a Postgres container |
| 3 | Raw http + in-memory | Map object | Simplest, passed all requirements |
 
Within-cell variance is high: sonnet+TS+beads doesn't converge on a single
architecture. Consider whether this cell needs more replicates or whether
the task spec is under-constrained.

Constraints:

• You MUST highlight cells where replicates disagree on framework/library/storage, not just scores.
• You SHOULD identify at most 5 "most divergent" cells — past that it becomes noise.
• You MUST NOT label divergence as a failure — sometimes it's the signal the experiment was built to find.

6. Surface shared issues

Aggregate findings across all runs. A finding kind that shows up in >50% of runs is a property of the task or the model, not of the individual run.

## Shared issues (appearing in ≥50% of runs)
 
| Finding | Runs affected | Severity |
|---------|---------------|----------|
| `skipped_test` | 28/37 | medium |
| `requirement_missing: R5 (pagination)` | 22/37 | high |
| `lint_warning: unused import` | 19/37 | low |
 
R5 (pagination) is either genuinely hard for agents OR the task spec
didn't emphasize it enough. Worth considering for a task-spec revision.

7. Write the report

Use the Output Format below. The report MUST link to individual evaluation.md files so readers can drill in.

Output Format

# Comparison: {experiment_dir_name}
 
Generated {timestamp} from {n} evaluated runs across {m} cells.
 
## Coverage
 
-Cells with ≥1 evaluation: {n}/{total}
-Runs evaluated: {n}/{total}
-Failed runs: {n}
-Missing evaluations: {list any}
 
## Per-cell summary
 
| Cell | n | Req pass | Test pass | Build fails | Shape diversity | Link |
|------|---|----------|-----------|-------------|-----------------|------|
| ... | 3 | 8.7±0.6 | 12.3±0.5 | 0 | 1 | [evals](runs/...) |
 
## Factor effects
 
### `{factor}`
...
 
## Qualitative divergence
 
...
 
## Shared issues
 
...
 
## Links
 
-ANOVA / effect sizes: `reports/analysis/`
-Per-run evaluations: `runs/<cell>/rep<N>/evaluation.md`
-Raw findings: `runs/<cell>/rep<N>/findings.jsonl`

Constraints Summary

• You MUST read evaluation reports, not re-evaluate. This skill is purely aggregation.
• You MUST NOT produce a single "winner" ranking. Retort is about factor effects, not tournaments.
• You MUST preserve replicate-level detail at least down to the per-cell table.
• You MUST cite specific runs by their archive path when surfacing qualitative claims.
• You MUST cross-reference retort analyze output when making causal claims about factors.
• Output MUST be markdown that renders in GitHub's viewer.

Troubleshooting

No evaluations yet

• Exit 0 with a single-line report: No evaluated runs yet. Run evaluate-run on each rep<N>/ first.
• Do not attempt to evaluate as a side effect — that's the user's / CLI's decision.

Evaluations with different schema versions

• Read what you can from each, note the schema mismatch in the "Coverage" section.
• Do not silently drop older evaluations — surface the problem.

`retort analyze` reports don't exist

• Proceed without them. Note in "Factor effects" that statistical backing is absent and the direction is qualitative-only.