kohulan/orthonym-skills/eval-loop

The evaluation loop

One iteration = one failure cluster. Stop after a small fixed number of iterations (e.g. 3) or when the goal threshold in your goals/bounds file is met, then report.

Copy this checklist and track your progress:

• [ ] 1. Measure: shipped and gates-off runs, both run ids noted
• [ ] 2. Diagnose: largest cluster, examples read, site proven by spy-site
• [ ] 3. Fix one cluster at the root
• [ ] 4. Re-measure: churn both ways, then the full eval
• [ ] 5. Gate: enforcer and pre-gate pass against a current baseline
• [ ] 6. Log: one row, negative results too; a fresh copy for the next iteration

Enforce your goals/bounds file mechanically, not by eye. After a re-run, evaluate the bounded contract with an enforcer script (or, at minimum, a checklist you actually run through every time) that derives every hard and relaxable bound from the run + gate artifacts and fails loudly on any hard-bound violation, naming the bound and both values:

<your-goals-enforcer> --run <runs-directory>/<id>.json --gate <gate-verdict-file>

A hard-bound violation (e.g. a wrong-structure/wrong-prediction rate rising, a protected-set regression, or an exact-match requirement breaking) means revert, do not trade: if a change makes a hard bound worse, undo the change, full stop, no matter what else it improved elsewhere. Write that rule down once in your own project's docs (an eval/LOOP-CONTRACT.md or equivalent) and read it alongside the goals/bounds file, so a fix that trades away a hard bound for a soft one is never mistaken for progress. The failure curriculum for the next iteration should be minted from the run or a wider audit — never from the held-out split or the protected set.

This is a developer workflow. Do not implant any part of it into the product/library itself: no self-correction loop, no retry ladder, no agentic behaviour inside the inference path just because your dev loop uses one. If your pipeline already calls an external validator at inference time (e.g. a round-trip parser gate), that is an existing design decision — this loop is not a licence to add more inference-time machinery.

Per iteration

1. Measure. Use the run-eval skill (or your project's equivalent). Run the shipped config for the headline number, and an ungated/diagnostic config for a clusterable run:

<your-eval-command> --split <dev-split> --tag "iter-N-before"
<your-eval-command> --split <dev-split> --gates off --tag "iter-N-before-diag"

2. Diagnose. Use the cluster-failures skill (or equivalent) on the diagnostic run. Take the largest cluster, but read its examples first: rows that share a surface feature need not share a cause (cluster-failures explains why).

Then prove the code site runs for the target rows with spy-site (or equivalent) before editing it; a site named in a comment, roadmap, or design doc is often called zero times. If it records zero calls on the target rows, return to Step 2 with the next site or cluster.

3. Fix at the root. Follow your project's fix-methodology doc if you have one (the governing rule, restated: build the whole class of cases correctly, or fail closed on the whole class — never a regex post-processor, never a per-case special case, never a string rewrite bolted onto an already-generated output).

Removing a wrong output is not automatically progress — check what the pipeline actually emits afterward, not just that the bad path stopped firing. A guard that blocks a fabricated output can unmask a different, equally wrong path underneath it (seen repeatedly across real projects: a rejected bad prefix silently drops atoms instead of refusing; an over-broad refusal turns a correct answer into a merely-valid but non-preferred one; denying one bad case falls through to an even more generic, wrong default).

4. Re-measure, cheaply. If your project has a cheap re-measurement tool — one that re-runs the pipeline but only re-invokes an expensive external validator for outputs that actually changed since the baseline run — use it:

<your-cheap-remeasure-command> --baseline <iter-N-before run id>

This should print churn in both directions (how many rows got better, how many got worse), so a "net +0" result that actually moved many rows each way stays visible instead of looking like nothing happened. Then run the full eval command for the iteration's headline number.

5. Gate. A change must not regress your protected-set metric: run the fast pre-gate (see the run-gate skill) and read the actual pass count against a current baseline — a bare PASS verdict is not sufficient if the baseline it's compared against is stale. If a hard bound got worse or the gate fails, revert and return to Step 3.

6. Log. Append one row to your project's eval log: iteration number, cluster worked, run ids before and after, the delta on every tracked bound, the commit, and one line on what actually changed. Log a negative result too — an iteration that moved nothing is one of the most useful entries in the log, because it stops the next session from repeating the same attempt.

Bounded objectives — maximise one, bound the rest

Per iteration, maximise one objective and hold every other tracked number as a bound, relaxed only deliberately and visibly (bounded-goals writes that contract).

Example table (substitute your project's actual metrics — a property predictor might use MAE-on-holdout as the objective with calibration/coverage as bounds; a retrosynthesis model might use top-k accuracy as the objective with route-validity as a bound):

Bound a cheap proxy like output length; never make it an objective. Length pressure truncates outputs into wrong ones (caffein for caffeine) that still pass a validity check.

Plateau trigger — hand it to a fresh agent

On an iteration that moves the objective by zero, hand the next step to a fresh context instead of iterating in the one that plateaued: a reviewer role analyses the code and writes an improvement plan to a file, and a fresh implementer executes it. The reviewer role should also hold sole authority to edit your goals/bounds file and this loop's own rules; an implementer that can move its own goalposts mid-iteration is not measurable.

A zero-movement iteration on your headline objective is not automatically a failure — check which bound the work actually targeted. A batch of fixes that holds the headline metric exactly flat while clearing a different bound (e.g. a spelling/formatting-layer correctness class that a round-trip-validity check literally cannot see) is real progress on that bound's axis, not a plateau. Confirm which axis the work was on before calling anything a plateau.

Rules that make the loop mean something

• One cluster per iteration. Two unrelated fixes in one iteration destroy the

attribution the next decision depends on.

• Never touch your held-out split unless the user asks in so many words. It exists

to be spent exactly once, at the end. Optional guard: the opt-in guard-holdout hook (hooks/guard-holdout.py in this plugin; setup in hooks/README.md) stops a held-out run until the user has agreed.

• Never re-sample a split. Split files are the contract; a good harness records

their content hash and refuses to compare runs across a hash change.

• Watch coverage and precision together. A headline correctness metric rising while

precision-on-emitted falls usually means abstention/coverage widened, not that outputs got better — especially when the headline metric is reference-free and measured identical whether validity gates are on or off, in which case a move that only shifts coverage/precision changed honesty, not correctness.

• You may run alongside other read-only measurement work if nothing writes the same

gate/verdict artifact; budget shared resources as run-eval says.

Stopping

Stop at a small fixed number of iterations (e.g. 3) or when your headline metric clears its threshold in the goals/bounds file. Then report: the trajectory across every tracked bound, which clusters were worked, which were tried and refuted, and the next-largest remaining cluster. If several iterations moved nothing, say so plainly — that's a finding about where the real difficulty is, not a failure to report.