boshu2/agentops/skill-eval

Skill Eval

Answer one named maintenance decision: retain, revise, remove, or insufficient evidence. Choose the measurement that can answer that decision, use the caller's accepted cases and resource envelope, make one scoped recommendation, and stop. A completed evaluation does not require a positive difference.

This is an optional specialist: the selected runner owns execution and bounds, and its native results own the measurements.

Rules that decide the answer

• Count every attempt. Keep failed, crashed, interrupted, blocked, abandoned,

missing and infrastructure-invalid attempts in the all-attempt accounting. A rerun adds an attempt; it never overwrites the one that failed. A run that shared its host with other work still counts for completion and tokens; keep its duration, labeled as measured on a shared host, rather than dropping it. Its timeout is not by itself a skill failure; when timeouts fall unevenly between arms, report them as a possible skill effect, not infrastructure.

• Vary one thing. Equalize instructions, tools, environment, model and

effort across arms apart from the intended variable. If one arm's task prompt repeats the skill's direction, attribute the result to the combined instructions, not the skill alone. When you control launch order, alternate arms run by run, swap which goes first in each pair and rerun a dead run in its own slot; one whole arm and then the other turns load, rate limits and time of day into an apparent skill effect.

• Confirm the skill loaded. Before reading a zero or small delta as no

benefit, check each treatment run for the skill actually being loaded or injected. A run where it never loaded measures routing, not content. When the act depends on a rule kept in a reference, also check that the run opened that file; a run that did not still counts; read its miss as the body's pointer to the reference failing, not the reference's rule, and report how many treatment runs opened the file. The behavioral probe injects only SKILL.md or a prelude, so it cannot answer this. Likewise, a case the no-skill arm already passes nearly in full has no headroom: report no headroom, not no value, and write a harder case (seeding).

• Calibrate the grader. Before trusting scores, confirm the judge or

discriminator passes a response that plainly meets each criterion and fails one that plainly does not. A weak judge can fail correct responses wholesale.

• Small samples are directional. Report uncertainty with every difference.

A difference without it shows neither benefit nor equivalence, and a zero-crossing interval is not equivalence. For two unpaired success rates, give each arm a Wilson interval, the difference with a Newcombe hybrid score interval and Fisher's exact test, not a normal approximation, which breaks at 0/n, n/n and a handful of runs. AgentOps ships no tool for this; the stock calls need the third-party scipy and statsmodels Python packages: scipy.stats.binomtest(k, n).proportion_ci(method="wilson"), statsmodels.stats.proportion.confint_proportions_2indep(k1, n1, k2, n2, method="newcomb") and scipy.stats.fisher_exact. If they are absent, say so; do not fall back to a normal approximation. Beside an inconclusive result, state the runs per arm needed to detect the smallest effect that would change the decision, and return insufficient evidence; running them is a new measurement with its own fixed stop and the caller's authorization, never an extension of this one.

• Fix the stop before running. Do not add trials until the result turns

positive, remove losing observations or relax acceptance.

• Know the noise before cutting. Before trimming or retuning skill text for

a new model, measure how much identical arms differ on the same cases. A cut must clear a bar fixed before the run, above that spread. A static read of the text is not a behavioral result. If a run's output was lost, a rerun is a new sample that spends the same limits; it does not restore the lost one.

Choose the question

Start from the caller's intended decision, not a mandatory quiz. For a behavioral question, name one observable action (a file written, tool used, criterion rejected); a belief such as “understands validation” needs translation into an action. For coding or memory questions, name unchanged task acceptance and the maintenance choice.

Prefer cases drawn from real use when authorized session history exists: a caller correcting, redoing or overriding the result right after the skill ran is the clearest sign of a miss. Label each case as drawn from use or written from the skill's own text; cases written from the text test what the author expected, not what users hit. With no usable history, say so rather than presenting text-derived cases as representative.

A skill whose result depends on the caller's answers, such as an interview, is not measured by its first turn; never run it where its questions reach a real person. Measure the dialogue as in dialogue skills.

Runners

Select the intersection of the caller's runtime authority, an available adapter and the named question. A runner's presence grants no runtime permission. If no route meets all three, return insufficient evidence with the missing route; do not automatically substitute another runtime or build a replacement runner. Forced SKILL.md injection, natural-request routing and installed-package coding comparisons answer different questions.

When Claude Code is explicitly authorized and its evaluator is available, claude plugin eval <plugin-path> --model <id> is an optional route. It runs the cases in the plugin's eval directory (evals/ by default) with the plugin and, by default (--ablation with-without), without it, scores each response with the case graders (LLM graders use --judge-model, default haiku) and reports the score delta. --runs sets repetitions per case, --max-cost-usd caps spend and --json writes per-run results. The model decides whether to load each skill, so the delta mixes routing with content. By default it also publishes its HTML report (prompts, responses and verdicts) to claude.ai and writes results under the plugin's eval directory: pass --no-publish, and point --output-dir, --json and --report at caller-selected storage. Confirm flags with claude plugin eval --help. In an AgentOps checkout, run the suites as `evals/plugin-eval/README.md` shows and grade with its grade.py: the default Haiku judge failed every criterion of correct validate responses there.

In an AgentOps source checkout, scripts/probe-skill.sh runs small behavioral probes through the Codex adapter (scripts/lib/codex-exec.sh), only when Codex and its isolation support are available and authorized. It injects the exact SKILL.md bytes (or a declared prelude), so it measures the text's effect on one act, not routing or an installed package. Neither runner's result substitutes for the other. Commands, replay limits and headroom classes are in behavioral probes.

The probe runner, evals/, the probe gates and every github.com/boshu2/agentops link in this skill are AgentOps source checkout only; installed skills do not ship them. Elsewhere, use the caller's authorized existing runner if it supports the question and record its identity and limits; with none, report the measurement unsupported rather than installing or starting a replacement.

Procedure