jordanhubbard/theseus/qualify

Qualify a gold-set family

Qualification is a research protocol, not the shipping product.

When to use

• A family already has characterized authority + public oracle + held-out oracle + implementation
• You are running tools/qualify.py or interpreting receipts under reports/qualification/

Hard rules

1. Never set qualified: true unless independent_generation is true (two empty-workspace gens, different impl hashes, both pass public + held-out + isolation + held_out_guard).
2. Do not copy factory theseus_* zero-arg wrappers. Gold-set oracles call the public API with arguments (theseus_json, theseus_*_q).
3. Do not put held-out vectors in zspecs/ public files or in synthesis prompts.
4. Keep gold/<family>/package.md qualification: none until dual-generation actually happens.
5. If ≥5 families are attempted and fewer than half qualify, do not advertise replacement. Characterization remains the product.

Commands

python3 tools/qualify.py <family>
python3 tools/qualify.py <family> --run1 <gen-a-pkg> --run2 <gen-b-pkg>
python3 tools/qualify.py --all
make qualify-check
python3 tools/lint_gold_wrappers.py

--run1 and --run2 are two external empty-workspace package directories (each contains the implementation). Both are required before a receipt can set qualified. --second-impl only compares a second tree with the in-repo implementation and cannot qualify. A single-family run rewrites that family's receipt and rebuilds summary.json from every receipt on disk.

Receipts

Each family writes reports/qualification/<family>.json. Summary and kill-gate bit: reports/qualification/summary.json.