jayfarei/opentraces/verifier-creator

OpenTraces Skill Verifier

A skill verifier is a calibrated rubric, not a checklist. The reframe that governs everything here:

A trace is evidence (what the agent did + the git lineage of what persisted), never the

verdict. The verdict — "was this skill used effectively?" — lives in a rubric you author

in conjunction with the skill's own goal. Calibration is trust: a rubric earns the right

to feed reward only when its verdicts demonstrably separate effective from ineffective use

against human gold and the tamper-resistant git signal. Until then it is BLOCKED, never a

silent pass.

Trust boundary (read first)

• The agent PROPOSES a rubric (criteria + evidence bindings + judging questions).
• The factory SCORES mechanically. A rubric can declare WHAT to judge; it can never set

reward/gate/split/value (the validator rejects those keys). Verdicts are computed from EVIDENCE, never from the skill text being optimized — so marker-stuffing has no channel.

• A human APPROVES promotion (always manual_required_default_off).

Two modes

AUTOVERIFY is the fast path: the agent reads the skill's goal, self-aligns a marker-structured rubric, judges it, and calibrates. Because self-alignment + self-judgment has no human anchor, its ceiling is provisional_weak_only — it may feed reward only when a deterministic criterion separates the external git/committed signal, and it can never reach calibrated (self-judged criteria earn no trust). Use it to bootstrap and to triage.

MANUAL lifts the ceiling: in one sitting you co-establish the desired outcome, edit the draft criteria, and label a handful of example/counterexample traces. Those labels are the gold that unlocks calibrated. The alignment session is where the labeling worklist happens — not a separate chore.

Criterion vocabulary

Each criterion has a judge_method:

• deterministic — a data-only DetectorSpec read off evidence (cheap, tamper-resistant). The

un-self-gradeable floor: every rubric needs ≥1 deterministic criterion carrying ≥20% of total weight, firing on <95% of traces and not presence-only.

• agent — the in-loop agent (you, on the user's subscription) judges an evidence-blind

packet locally and posts a verdict. The default for semantic criteria ("are the findings substantive?"). Earns weight only after it is judged AND calibration shows it discriminates.

• human — gold labels; the calibration anchor and the only writer to the gold ledger.

A direction: negative criterion's presence is a failure (e.g. "claimed success but nothing committed") — this is how you manufacture a discriminating negative class from data.

The loop (both modes)

1. ORIENT — read_skill_definition(skill): read the TARGET skill's goal. In MANUAL mode,

ask the user: "what does an effective `<skill>` invocation achieve — in outcome terms, not the steps it follows?"

1. `list_candidates(skill)` — evidence + which markers are deficient + calibration

feasibility (n_weak_neg, n_human_labels). It says plainly when negatives are too few.

1. `get_skill_examples(skill, episodes=...)` — real example vs counterexample trace refs.

Pull the windows and READ what the traces did: opentraces trace slice <id> --template bursts --json, opentraces trace get <id> --json, opentraces trail track <id> --json (survival, best-effort).

1. Author the rubric:

• AUTOVERIFY: autoverify(skill, episodes=..., records=...) self-aligns + calibrates in one

call. Read result.calibration.status.

• MANUAL: align_session(skill, ...) scaffolds the desired-outcome prompt, an editable draft

rubric, and the traces to label. Edit criteria, then author_rubric(spec) (validates; rejects reward/gate keys, ungrounded semantic criteria, missing floor, permissive floor).

1. Judge the semantic criteria (agent): for each agent criterion × trace,

build_judge_packet(...) → reason over the bound evidence (NO skill text) → post_verdict(..., evidence_quote=<verbatim span>). Judge the OUTCOME, not step-conformance. Abstain if the evidence is insufficient.

1. Label (manual): record_human_label(package_dir, ..., label=0|1, human_confirm=True) —

the only writer to the gold ledger, requires an explicit human keystroke. Label the example/counterexample traces from step 2 together.

1. `calibrate_rubric(rubric, episodes=...)` — per-criterion precision/recall/discrimination,

AUC vs gold, Spearman vs the weak class, the adversarial gate. READ THE STATUS:

• calibrated — verdicts separate good from bad vs gold; may feed reward.
• provisional_weak_only — discriminates the weak class but no gold; reward usable, flagged,

recommended=False.

• blocked_* — STOP. The blocker names the remedy (gather N more negatives, redesign a

non-discriminating criterion, fix an inverted one). Never report a passing score on a blocked rubric.

1. `score_rubric` / present for approval — only on calibrated/provisional. The headline

shows AUC + per-criterion discrimination + adversarial PASS + status, NOT a bare Dsel 0→1. Present for HUMAN APPROVAL. Do not promote.

MUST NOT

• Set reward/gate/split/value/calibration/recommended in a rubric (rejected).
• Judge the optimized skill text (judge the evidence); judge step-conformance instead of outcome.
• Write the gold ledger from the agent path; mark recommended or promote; author a broad/

degenerate rubric to pass the gate (hard-gated).

• Present a green headline for a blocked or merely provisional rubric. Autoverify is never

calibrated and never auto-recommended without human gold.