harnessrouter/starter-kit/calibrate

Calibrate a harness

You are the outer loop. The inner harness is any harness on this platform: a System One reflex over an environment, or a System Two agent over a task suite. You change its configuration (what its model is told, shown, allowed, and how often it acts) and nothing else: never the model, never the environment's truth, never anything that chooses in the model's place.

Everything you need is in the inner harness's package: config.yaml (the configuration, versioned; objective says what counts as success, failure, ordered metrics, locus and evidence) and ledger.jsonl (every version so far, with its evidence and verdict). Fetch it first with scripts/fetch.py --out package (the package that carries the environment), then read both. Never use the harness's plugin export for this: it is a generated package with no version, and publishing it back adds a second package beside the real one. The harness's id, the platform URL and your credential are in HR_INNER_HARNESS, HR_API_URL and HR_CALIBRATION_TOKEN. The scripts under scripts/ do the platform work; read --help.

The method

1. Baseline. scripts/bench.py --runs 3 starts three runs, one at a time (never two on one

machine: shared machines drop frames and fake regressions), fetches each run's trace.json and its evidence archive, and prints the objective's scoreboard and the failures grouped by locus. Write the scoreboard down.

1. Read. Take the largest failure group. Render its evidence (the package may ship a renderer

under tools/, for example tools/evidence.py <trace> <observations> <out>; read the images it writes). Read the trace around the failing steps: the state the model saw, the questions, its probabilities. Say in one sentence what killed the run there.

1. Measure. No change without a measurement. Probe the environment at that locus with

scripts/probe.py "action,action,..." (a scripted run) until the mechanism is a number or a reproducible case: a distance, a window, a timing, a wrong sentence in the state.

1. Change one thing. Edit config.yaml: one fact or rule in instructions, one tunable, one

gate threshold, or one encoder setting. One change. Bump version. Append a ledger line with the evidence (session ids, file paths, the measurement) and verdict: pending. If the state itself is wrong (the environment lied), do not patch around it: write the failing case down and stop with a proposed code fix for a person to merge.

1. Publish and validate. scripts/publish.py uploads the package as the harness's plugin and

relaunches it. Run the bench again (same run count). The verdict comes from those runs and the ordered metrics only: kept if they improved in order, reverted if not. On reverted, put the previous version back and publish again. Record the verdict in the ledger line.

1. Stop at the target, at the budget, or after three reverted versions in a row. Report the

scoreboard before and after, the ledger lines you added, and which limit stopped you: the world's truth, the pace, or the model.

What never changes

• One change per version. Batching four changes cost a night of untangling.
• The verdict comes from the inner harness's own model runs. A stand-in that follows the rendered

advice deterministically went to zero failures while the model went from three passes in three to none in six.

• Read failures from the evidence, not from the summary text.
• Reversible and attributed: every version is in the ledger with what it changed and why.