skillberry-ai/cap-evolve/finalize

finalize — the one honest number

Optimization hill-climbs on val: every accept decision consumed val as a tuning signal, so by the end of search val is optimistic — it has been selected against. The number you report must come from data nothing was tuned against. finalize scores the run's best candidate on the sealed test split, once, and writes final.json. That file is the run's result.

One finalize is two evals, not one

A bare test number cannot be defended — a reader cannot tell whether it beat the capability you started with. So one finalize scores test twice: the best candidate as tag FINAL, and the untouched seed candidate as FINAL_seed (harness.finalize). final.json therefore carries test, test_baseline, baseline_id, and test_delta — the held-out improvement, which is the figure report, the dashboard, and the event stream all headline. If the best candidate IS the seed (nothing was accepted), the second eval is skipped and test_delta is 0 by construction.

So budget --n-trials 3 as 3 trials × 2 candidates × |test| rollouts — twice what the flag looks like it buys on a paid benchmark.

Both evals sit inside one attempt and neither is a selection event: the delta is reported, never chosen on. That is why the seal counts attempts, not evals.

The seal (why "exactly once")

The instant test informs any choice — picking between finalists, "double- checking" a low number, re-running until it looks better — it stops being held out, because each peek is a selection event that pulls the number from an unbiased estimate toward an optimistic fit metric (references/concepts.md).

cap_evolve enforces this in three parts (rundir.py:358-407), and the split between them is the whole design:

• reserve — every split="test" eval first checks the seal without burning

it, so no phase other than finalize can reach test at all.

• commit — the seal burns only after final.json is written, so a finalize

that dies before scoring leaves it unused and is honestly retryable. A transient crash must not destroy a run's headline number.

• attempt guard — seal-on-success alone cannot tell "crashed before scoring"

from "crashed after". A real run hit the second case: a finalize killed by a timeout had already scored test, the retry scored it again, and the reported headline was that second look. begin_test_attempt refuses a retry once test rollouts exist on disk, before anything is spent.

The seal refuses that mistake by default; it is not unbypassable. CAPEVOLVE_ALLOW_TEST_RESCORE=1 is a deliberate opt-in override (rundir.py:166). Its own message promises the use "is recorded in the run" — nothing records it (issue #341), so a run that took a second look currently looks identical to an honest one. If you set it, disclose it in the write-up yourself.

Corollary: all selection happens before finalize. Choose the single best candidate on val, then finalize it. Finalists that genuinely need comparing get compared on val — never on test.

If finalize refuses

A TestSealError is three situations with three different right moves. Tell them apart from <run>/rollouts/test/ and test_used in splits.json:

Never delete test rollouts or edit splits.json to get past the error — that manufactures a clean-looking number from a split that has already been seen.

Dual-mode

This phase runs two ways from the same SKILL.md: standalone as the slash command /cap-evolve:finalize (the argument-hint shows its run.py args), and orchestrator-callable — cap-evolve run / the orchestrate skill invokes the same scripts/run.py headlessly and threads the run dir between phases.

How to run

python scripts/run.py --run-dir .capevolve/run_XXXX --project .capevolve/project --n-trials 3

Multiple trials give the headline an honest stderr and a pass^k reliability figure instead of one noisy point. Under cap-evolve run the count comes from num_trials in capevolve.yaml and defaults to 1 — set it to ≥3, or the orchestrated headline ships with stderr 0: the exact single point this warns against. If the split was configured with no holdout (test == train/val) the number is a fit metric, not a held-out result; the dashboard flags it, so say so in the summary too.

Then read the result instead of just filing it: test ≈ val means the val gain generalized, test ≪ val means search overfit val — a real finding, not a reason to re-score.

References

• references/concepts.md — why each peek biases the estimate, the

train-fits / val-selects / test-estimates rationale, no-holdout runs, and how this maps to public benchmark protocol, with sources.