typefox/agent-skills/skill-evals

Evaluating a skill with the eval loop

A skill is a bet that some instructions make an agent's output better. The loop is how you collect on the bet: run real prompts with the skill and against a baseline without it (or against the previous version), grade both against the same bar, and read the delta — what the skill costs in time and tokens versus what it buys in pass rate. Repeat until the delta stops moving.

This skill is the spine of that loop — the order of operations, the two gates that guard it, and the two points where only the user can supply what you need. The mechanics — exact file schemas, the grader and analyzer subagents, the aggregation script — live in skill-creator, which this skill drives rather than restates. When you need a schema or a command, follow the pointer into skill-creator.

One deliberate deviation from skill-creator. skill-creator surfaces results through an HTML eval viewer plus a generated benchmark.md and review.md. skill-evals does not: it runs skill-creator's scripts only to process and aggregate the JSON (grading.json, benchmark.json, timing.json), then consolidates everything human-facing into a single `REPORT.md` (Activity 7). Do not launch the eval viewer or rely on the scripts' Markdown/HTML output — REPORT.md is the one artifact the user reviews.

Preflight — clear both gates before any activity

Gate 1: skill-creator must be available

Locate the installed skill-creator skill and read its SKILL.md. Note its directory as SKILL_CREATOR; every mechanic below is addressed relative to it — SKILL_CREATOR/references/schemas.md, SKILL_CREATOR/scripts/, SKILL_CREATOR/agents/.

If skill-creator is not installed, stop here. skill-evals is an orchestration layer and cannot run without it. Tell the user to install it (it lives in the anthropics/skills repository) and re-run. Done when SKILL_CREATOR resolves to a readable directory, or you have stopped with that instruction.

Gate 2: runs must be isolated — and provably so

Every eval run must start from clean context. The clean way to get isolation is a subagent: a child task with fresh context and one dedicated prompt.

But "I told the subagent not to read the skill" is not isolation you can trust. A baseline that loads the skill under test, the with-skill version, the live copy instead of the snapshot, or anything outside its allowlist inflates its score and silently shrinks the delta — sibling skill files and parent-thread instructions are often one Read away. You must be able to verify what each run loaded, not assume it — and since runs read files through native tools like Read and Glob, the mechanism has to capture those tool calls, not just commands. Pick the strongest your harness supports:

1. Allowlist enforcement (preferred). If you can sandbox or read-scope a subagent to an allowlist of paths, do it: each run gets only what its configuration permits — for the baseline, that configuration's skill set (none, the snapshotted old version, or named helper skills if any) plus eval inputs and its output dir; for with-skill, exactly one version of the skill under test plus inputs and output dir. This makes leaking impossible rather than merely discouraged, so nothing needs auditing afterward — the sandbox itself is the proof.
2. Deterministic tracing (second). If you can't restrict paths but the harness records a machine-readable trace of every tool call a run made — file-access tools included — audit that trace against the allowlist once the run lands. The harness captures it, so it costs the run nothing and the agent can't edit it away. Where it lives:

• Claude Code persists each subagent's transcript at <session-dir>/subagents/agent-<id>.jsonl (you hold the agent id from the spawn) — or nested one level deeper, under subagents/workflows/wf_<run-id>/, when the runs came from the Workflow tool rather than a direct spawn. Scan it for Read / Edit / Write / Glob / Grep tool_use entries and every `Bash` command string — a cat/find/ls/grep -r inside Bash is just as real a read as the dedicated tools and is easy to miss if you only scan typed tool calls — and confirm every path is on the allowlist. Generating through Workflow (e.g. to run generation on a cheaper model than the one orchestrating) has its own isolation quirks on top of this; see references/claude-code-workflow-tool.md.
• Codex emits the equivalent when launched as codex exec --json — read the JSONL event stream and check its file-access events against the allowlist.
• Cursor exposes this only if the installed version provides agent hooks (log tool calls through them); absent that, its history lives in undocumented state.vscdb SQLite blobs too version-fragile to trust, so drop to self-report instead.

1. Self-report (fallback). If neither holds, require every run to log the resources it loaded: instruct the subagent to append each skill / spec / input file to <run-dir>/access-log.md as it reads the file — not only at the end, because a file read right before a Write can slip past a summary log — and to leave its metrics.json tool-call counts as corroboration. After the runs you audit each log against that run's allowlist. It is last because it adds work to the run and trusts the agent to report honestly against itself.

One channel no tier catches: harness-injected project memory. Harnesses inject the repo's root instruction files (AGENTS.md / CLAUDE.md and their hierarchy) directly into subagent context at spawn — in Claude Code this reaches every custom/general subagent and every Workflow-spawned agent (only the read-only built-in Explore and Plan agents skip it), and no opt-out exists. The injection is not a tool call, so sandbox records, transcript traces, and access logs all come back clean while every run silently carries repo instructions that bias it — repo conventions at best, and at worst notes about the eval fixtures themselves ("files under evals/ are intentionally broken"), an answer key handed to the baseline. The content is cached when the orchestrating session starts: moving the files aside mid-session does not help (verified — a subagent spawned while both files were off-disk still quoted them verbatim). The check is cheap: when the eval process starts, look for root instruction files (AGENTS.md, CLAUDE.md, or tool-specific equivalents). If they exist, stop and have the user remove them and start a fresh eval session — they are restored from git afterwards (git checkout -- AGENTS.md CLAUDE.md). One caveat closes the cache loophole at no extra cost: absent files prove nothing if the session started before they were removed, so also confirm your own context carries no injected root-file content — if it does, this session predates the removal and must be restarted. Record the check's outcome alongside the rest of Gate 2's clearance.

Escape hatch: restarting is not the only remedy — the out-of-process headless run mechanism below sidesteps the cached injection entirely, so a contaminated orchestrator can still run the eval; see the run-mechanism choice for how and for the conditions that keep it honest.

Write the per-run allowlist down before launching — which skill version (or none), which input files, which output dir. That allowlist is what the isolation verification (Activity 2) and the isolation assertions (Activity 3) check against.

Two run mechanisms can satisfy the gate — choose one explicitly and record the choice with the rest of Gate 2's clearance:

• In-process subagents (a Task / subagent tool that spawns fresh-context children, or Claude Code's Workflow tool): each run is one subagent, scoped or logged as above. The default when the orchestrating session started clean: it gives you parallel fan-out in one turn and, through Workflow, cached targeted reruns (see references/claude-code-workflow-tool.md). Its limits are the flip side of being in-process: every child inherits the session's cached project memory (the channel above — prevention only, never verifiable), and Workflow surfaces no trustworthy per-run token count.
• Out-of-process headless sessions (Claude Code: claude -p; Codex: codex exec): each run is its own fresh process, which assembles project memory from disk at its own start — so with the root files off disk nothing is injected, whereas in-process children always inherit the orchestrating session's cached copy. That makes this the escape hatch for a contaminated orchestrator (the only option short of restarting), and unlike the in-process flavor the project-memory channel is verifiable, not merely prevented: injected content would appear verbatim in the first user turn of each child's own persisted transcript — for Claude Code under ~/.claude/projects/ (keyed by the run's working directory), the same transcript the tier-2 trace audit reads — so grep the child transcripts for distinctive root-file content and record the result with Gate 2's clearance. Two conditions keep the guarantee honest: the files must be absent along each child's entire cwd→root walk and at the user level (e.g. ~/.claude/CLAUDE.md), and they stay off disk until the last child — retries and graders included — has finished. Headless is a legitimate choice from a clean session too: pick it when the cost side of the delta matters (claude -p --output-format json returns harness-measured tokens, duration, and cost for timing.json — which Workflow cannot) or when runs need a different model (--model). Its costs: you script the fan-out and retries yourself, and there is no cache/resume for cheap targeted reruns.
• Neither available (no subagent tool, no headless CLI): do not quietly run the prompts inline — your context already holds the skill, so an inline "baseline" is fiction. Instead, stop the automated loop and hand the user a fallback: generate the full set of run prompts (one block per run, contents in Activity 2) for them to paste into separate fresh sessions, then collect the outputs into the run directories and resume at grading.

Done when you have chosen an isolation mechanism (allowlist, trace, or self-report), written the per-run allowlists, chosen and confirmed the run mechanism (subagents spawn, or the headless CLI runs), and cleared the project-memory channel (no root instruction files on disk, none injected into your own context — or the escape hatch taken) — or switched to the copy-paste fallback and told the user.

Record gate clearance in the workspace before Activity 2 runs — a short methodology-and-isolation.md (or note in the first summary.md) under the eval output root, naming the resolved SKILL_CREATOR path (Gate 1) and the isolation mechanism + per-run allowlists (Gate 2). Without this paper trail, later grading cannot tell whether preflight actually happened or whether a run was contaminated.

Pick an entry point

The loop runs end-to-end or à la carte — support whichever the user asked for. Find their state and jump in:

• "Evaluate / test my skill", "does this skill actually help?" → run the whole loop from Activity 1.
• Test cases already exist (an evals/evals.json) → start at Activity 2.
• Runs already produced outputs → start at Activity 3 (assertions) or 4 (grading).
• A benchmark already exists → Activity 6 (analyze) or 7 (human review).
• "Make it better from these results" → Activity 8.
• A single activity named outright ("just grade these", "aggregate the results") → do that one and stop.