adrianco/retort/update-optimal-blog

Update optimal-blog.md

Overview

optimal-blog.md records what to run today, per language and task size. Its data tables are not hand-written — they are generated from master.db by the retort report optimal subcommand (code in `src/retort/reporting/optimal.py`) and live between <!-- GEN:<key> START/END --> markers. This skill is the safe procedure for refreshing them: check the data before you trust it, regenerate, verify the round-trip, then fix any prose whose numbers moved.

The order matters. master.db does not record the full stack/config (see the health gaps below), so a blind regenerate can silently publish wrong numbers. Always run the health check first and stop if it reports anything new.

Why per-language, not aggregates

A single cross-language reliability number is misleading — it blends a stack's strong languages with its weak ones (local Qwen passes Python/Go but fails Rust; Opus 4.8 dips on Java). The generator's centrepiece is the per-language success-rate matrix; the leading-stacks routine aggregate is explicitly labelled the least-useful number. Keep it that way — do not "promote" an aggregate back into the recommendation.

Steps

1. Health-check master.db FIRST (gate)

retort report optimal --health

Compare against the known, accepted gaps (already documented in the blog's Keeping this current section):

• ⚠️ No sampling columns / max_context_tokens unpopulated — the qualified config is

curated in FEATURED_STACKS, not filtered from data.

• ⚠️ ~250 rows have a blank model (local provenance bug) — attributed by experiment slug.
• ⚠️ experiment-11, experiment-29 not ingested.

You MUST STOP and surface to the user if the report shows anything beyond those:

• "Unmapped model strings" — a new model appeared that no featured/legacy entry

covers. Decide whether it's a new featured stack (add to FEATURED_STACKS) or legacy (add to KNOWN_NONFEATURED) before regenerating. Publishing without deciding would drop it silently.

• New experiment dirs not in master.db — results exist on disk but aren't ingested;

the refresh would omit them. Re-ingest first, or note the omission to the user.

• A jump in the blank-model count — the harness may have regressed; a run recording no

model is invisible to the tables. (The fix landed in src/retort/playpen/runner.py stack_metadata(); if new blanks appear, that path is being bypassed.)

2. Regenerate the tables

retort report optimal --write optimal-blog.md

This splices every GEN:* block (leading stacks, per-language matrix, per-language winners, prompt method). It only touches text between markers.

3. Verify the round-trip

retort report optimal --write optimal-blog.md   # run twice
git diff --stat optimal-blog.md

A second --write MUST produce no further change — the tables are idempotent. If git diff shows table cells changed, that's the real new data; if it shows nothing, the blog was already current.

4. Reconcile the prose

The generator owns the tables, not the sentences around them. After a refresh, read the diff and fix any prose that quotes a number that moved:

• The bullets under Leading stacks (e.g. "Opus 4.8 ~0.59 on hard").
• The per-language recommendation table (the curated one with the prompt/testing column)

and its †/‡ footnotes — reconcile its picks against the generated matrix and winner table. A cell that flipped qualified↔unqualified (e.g. a language gaining local support) changes the recommendation.

• The decision procedure list.
• The language split sentence if a language moved between local and cloud.

You MUST NOT edit numbers inside the GEN markers by hand — re-run the generator instead.

5. Report

Tell the user: which table cells changed, any prose you reconciled, and — if step 1 found anything — what you stopped on. If nothing changed, say the blog was already current.

Constraints Summary

• You MUST run --health and clear it against the known gaps before --write.
• You MUST NOT hand-edit between GEN:* markers; the generator is the source of truth.
• You MUST NOT introduce or re-elevate a cross-language aggregate as a recommendation —

per-language success rates are the point.

• You MUST verify idempotency (step 3) before considering the update done.
• A new model string or un-ingested experiment is a STOP-and-ask, not a silent skip.

Troubleshooting

`--health` reports an unmapped model — add it to FEATURED_STACKS (with a models list and short column name) if it should appear in the blog, or to KNOWN_NONFEATURED if it's legacy/a serving variant. Re-run health until only the accepted gaps remain.

Blank-model count grew — the harness stopped recording model for some runs. Check stack_metadata() is still called by every runner's provision() (local_runner, metaharness_runner, docker_runner); a new runner or a bypass would reintroduce the original bug. Do not paper over it in the generator.

A local language's number looks too low — confirm the curated selection in FEATURED_STACKS still points at the tuned-config experiments, not an all-experiment average (which includes early bad-config runs and understates the tuned reality).