heshamfs/materials-simulation-skills/skill-evaluator

Skill Evaluator

Test whether a skill is correct, discoverable, and valuable — not just whether its unit tests pass. The harness is agent-agnostic: it drives whichever coding-agent CLI the user uses, because Agent Skills are portable across all of them.

When to use which layer

Three layers, increasing cost and fidelity (full rationale in references/methodology.md):

Always run Layer 1 (it's free). Add Layers 2–3 when you can run a coding-agent CLI.

Step 0 — pick the agent CLI

Ask the user which coding agent they use, then map it to an adapter id. Supported: claude-code, openai-codex, antigravity (the agy CLI that replaced Gemini CLI on 2026-06-18), cursor-cli, github-copilot-cli, amp, opencode, grok-cli. See the full matrix and auth in references/adapters.md, or run:

python scripts/agent_adapters.py list

Confirm the binary is installed and the auth env var is set (the matrix lists it). Before any real run, dry-run it to see the exact command:

python scripts/agent_adapters.py build <agent> --prompt "test" --workdir /tmp/wd

Step 1 — deterministic script checks (always)

python scripts/run_script_checks.py --skill <path-to-skill> --json

Runs the script_checks in the skill's evals/evals.json, executing each script and grading its --json output against machine-checkable assertions. Exit non-zero on any failure — safe for CI. If the skill has few/no script_checks, add them for every eval whose answer is computable (schema in references/schemas.md); this is the cheapest, most durable guard against doc↔code drift.

Step 2 — trigger / discovery eval

Does the description fire on the right prompts and stay quiet on near-misses?

# Dry-run first (prints the per-CLI commands, runs nothing):
python scripts/run_trigger_eval.py --skill <path> --agent <agent> --dry-run
 
# Real run with a labelled query set (~20: half should-trigger, half near-miss):
python scripts/run_trigger_eval.py --skill <path> --agent <agent> \
  --queries queries.json --runs-per-query 3 --json

Design the query set per references/methodology.md (positives + tricky negatives). Without --queries, the skill's eval prompts are used as should-trigger cases — add negatives for a real discrimination test.

Step 3 — output-quality eval (the with/without delta)

The headline measure: does an agent following the SKILL.md beat no skill?

# 1. Dry-run the plan (no tokens spent):
python scripts/run_quality_eval.py --skill <path> --agent <agent> \
  --workspace <skill>-workspace --dry-run
 
# 2. Real run: with-skill AND no-skill baseline, isolated clean dirs each:
python scripts/run_quality_eval.py --skill <path> --agent <agent> \
  --workspace <skill>-workspace --iteration 1 --json

This installs the skill into a temp project skills dir for the with-skill run, runs a clean baseline without it, and captures outputs/, response.txt, and timing.json per run.

Then grade each run against its assertions and write grading.json (references/grader.md — re-derive numbers, require concrete evidence, no partial credit, critique weak assertions). For mechanically checkable assertions, reuse Layer 1 rather than eyeballing.

Then aggregate into the benchmark with the delta:

python scripts/aggregate_benchmark.py <skill>-workspace/iteration-1 \
  --skill-name <name> --agent <agent> --json

run_summary.delta.pass_rate is the value of the skill. Surface patterns the averages hide (references/methodology.md): non-discriminating assertions, high-variance evals, time/token tradeoffs.

Then generate the review and put it in front of the user *before* you self-grade (a standalone HTML page — no server needed):

python eval-viewer/generate_review.py <skill>-workspace/iteration-1/benchmark.json -o review.html

It renders the with/without delta, per-configuration stats, and an expandable per-eval breakdown of each graded assertion (text, pass/fail, evidence).

Step 4 — iterate

Improve the skill from the signals (failed assertions, weak-assertion feedback, transcripts, human review), generalizing rather than overfitting, keeping it lean, explaining the why, and bundling repeated work into scripts. Rerun into iteration-<N+1>/ and compare. Stop when results satisfy the user, feedback is empty, or gains plateau. For "is the new version actually better?", use the blind comparison described in references/methodology.md.

Outputs to report

• Layer 1: checks passed / assertions passed; any doc↔code drift found.
• Layer 2: trigger pass rate (positives that fired, negatives that stayed quiet).
• Layer 3: with-skill vs. without-skill pass rate delta, plus time/token cost.

Reference files

• references/adapters.md — per-CLI headless command, skills dir, auth, caveats.
• references/methodology.md — the rigorous practices (read for non-trivial evals).
• references/grader.md — how to grade a run into grading.json.
• references/schemas.md — exact JSON shapes for every file.
• eval-viewer/generate_review.py — render a benchmark into a standalone HTML review.

Verification checklist

Do not report a verdict until each item that applies to the layers you ran is satisfied:

• [ ] Layer 1: ran run_script_checks.py --json, recorded the summary line (checks_passed/checks, assertions_passed/assertions), and confirmed ok: true (process exit 0) — a non-zero exit means doc↔code drift, not a passing skill.
• [ ] Layer 1: for at least one numeric assertion, re-derived the expected value by hand and confirmed the script's emitted value matches it (e.g. approx within the stated rel_tol/abs_tol) — not merely that the assertion's passed flag is true.
• [ ] Layer 1: recorded cases_without_checks; if any computable eval lacks a script_check, noted it as a coverage gap rather than treating the run as fully verified.
• [ ] Layer 2: ran run_trigger_eval.py with a labelled set containing both positives AND tricky negatives, and recorded the per-class pass counts (positives that fired at rate ≥ threshold, negatives that stayed below) — a positives-only run measures recall, not discrimination.
• [ ] Layer 2: used --runs-per-query ≥ 3 and recorded each query's trigger_rate; flagged any query whose rate sits near the --threshold as unstable rather than counting it as a clean pass/fail.
• [ ] Layer 3: ran BOTH with_skill and a without_skill baseline, then reported run_summary.delta.pass_rate (the headline value) with mean ± stddev — never an absolute with-skill pass rate alone.
• [ ] Layer 3: graded each run from the actual files in its outputs/ (re-deriving numbers / opening artifacts per references/grader.md), recorded concrete evidence per expectation, and put the outputs or benchmark.md in front of the user before concluding.

Common pitfalls & rationalizations

Security

Input Validation

• --agent is resolved against a fixed allowlist of known adapter ids/aliases

(agent_adapters.py); unknown values are rejected (exit 2).

• --skill must be a directory containing SKILL.md or the runners exit 2.
• script_checks operators and dotted paths are matched against fixed sets; no

user string is ever eval()'d or passed to a shell.

File Access

• The deterministic layer runs a skill's own scripts with the real interpreter and

reads only that skill's evals/evals.json.

• The quality/trigger layers create isolated working directories under a

user-supplied workspace, copy the skill into them, and write results there.

Tool Restrictions

• Bash: runs the harness Python scripts and the selected coding-agent CLI.
• Read/Grep/Glob: inspect skills and results. Write: scaffold workspaces.

Safety Measures

• No eval()/exec(); subprocess calls use explicit argument lists (never

shell=True); commands are built from the adapter spec, not string-concatenated.

• The trigger/quality layers pass each CLI's auto-approve flag (e.g.

--dangerously-skip-permissions), which runs the agent with reduced safeguards. Only evaluate skills you trust, ideally inside a sandbox/container. Always --dry-run first to inspect the exact command. Auth is read from environment variables, never passed as command arguments.

Limitations

• Layers 2–3 require a supported CLI installed and authenticated; otherwise use

Layer 1 only.

• Trigger detection is a cross-tool heuristic (did the transcript consult the

skill?); for the most precise detection on Claude Code, parse its stream-json tool-use events.

• Token accounting is best-effort — only some CLIs report usage in headless output.
• New CLIs (Antigravity, Grok) are medium confidence; verify flags with the

vendor --help and --dry-run.

Numerical and evidence limits

Quality and trigger runners require fresh workdirs, reject ancestor agent configuration, and use allowlisted environments plus private home/config directories. This is configuration isolation, not an OS sandbox. System-managed CLI skills/tools and live authentication still require separate qualification. Failed/timed-out executions are ungraded. Token usage remains null when unavailable; paired deltas require complete matched measurements. Trigger detection remains a transcript heuristic, not measured discovery accuracy without a live benchmark.