fastxyz/skill-optimizer/skill-optimizer

skill-optimizer

skill-optimizer is an eval workbench for agent skills. It runs a model in an isolated Docker /work directory, provides skills/references as normal workspace files, captures an agent trace, and grades deterministic local outcomes.

Use this skill as the source of truth for authoring eval suites in this repo. Detailed schema and patterns are in references/workbench.md.

Core Model

• A case is one user-like task plus one or more deterministic graders.
• A suite is a set of cases and OpenRouter models to run as a matrix.
• references are copied into /work before the agent starts; this is where eval skills live.
• The agent phase sees /work only. It cannot see /case, /results, graders, hidden answers, or hidden metadata.
• Cases can define mcpServers; these are exposed through a workbench mcp command during the agent phase.
• Graders run after the agent with /case, /work, and /results mounted.
• trace.jsonl is the debugging source for what the agent saw, said, and did.

Commands

Rules:

• Use only openrouter/... model refs.
• OPENROUTER_API_KEY is required for real model runs.
• run-suite uses models: from suite.yml; it has no model override flag.
• run-case can use its case model: or --model / --models.
• Docker image default is skill-optimizer-workbench:local.

Install This Skill

This repository ships one canonical skill at skills/skill-optimizer/SKILL.md plus plugin metadata for Claude Code, OpenCode, Codex, Cursor, and Gemini.

Install the skill for common agents with:

npx skills add fastxyz/skill-optimizer --skill skill-optimizer -a claude-code -a opencode -a codex -a cursor

Plugin entrypoints:

• Claude Code: .claude-plugin/plugin.json and .claude-plugin/marketplace.json
• OpenCode: .opencode/plugins/skill-optimizer.js
• Codex: .codex-plugin/plugin.json
• Cursor: .cursor-plugin/plugin.json
• Gemini: gemini-extension.json and GEMINI.md

Authoring Workflow

1. Create suite.yml with models, shared defaults, and inline cases or case paths.
2. Put the skill/reference material under references/; it will be copied into /work.
3. Write natural user tasks. Do not mention graders, hidden answers, /case, or eval internals.
4. Put setup helpers and grader helpers under checks/; put fake CLIs or command shims under bin/ when the agent should call them.
5. Add one or more graders per case. Prefer small deterministic graders over one broad grader.
6. Run run-suite --trials <n> and inspect suite-result.json, failing result.json, summary.json, and trace.jsonl.

Variables listed in env are forwarded unchanged into setup, agent, grading, and cleanup containers. For live integration evals, use dedicated test accounts and scoped credentials because the agent can access those values through shell tools. Treat trace.jsonl, result.json, grader evidence, stdout/stderr, and preserved workspace/ directories as potentially sensitive if an agent or grader prints or writes secret values.

Use mcpServers when the task should interact with MCP tools. For local servers whose source should stay hidden from the agent, put server files under the case mcp/ support directory and define mcpServices; Docker starts those as separate service containers and the agent only sees their HTTP MCP URL. Direct stdio mcpServers.command entries run inside the agent container and are only appropriate when the server implementation is intentionally agent-visible. Remote HTTP/SSE servers must be reachable from Docker. The workbench generates /work/mcporter.json with imports: [], so host/user MCP configs are not imported. OAuth/browser auth is not supported; use env/header credentials listed in env.

Prefer the real CLI/API/service when you do not know its internal behavior well enough to mock it faithfully. Mock only when you are sure the mock matches the real command surface, validation, outputs, and failure modes; otherwise the eval will measure the mock, not the skill. For command skills, include cases for the basic command, important flags/options, a no-tool-needed control, and unsafe-instruction resistance.

Minimal Suite

name: pdf-skill-eval
references: ./references
models:
  - openrouter/google/gemini-2.5-flash
env:
  - OPENROUTER_API_KEY
timeoutSeconds: 600
setup:
  - node $CASE/checks/create-inputs.mjs
appendSystemPrompt: |
  Keep task outputs at the top level of /work unless the user asks otherwise.
cases:
  - name: extract-pdf-facts
    task: |
      Read statement.pdf and write answer.json with the account, quarter, approval code, and risk flags.
    graders:
      - name: answer-json
        command: node $CASE/checks/extract-pdf-facts.mjs

Directory Layout

my-eval/
  suite.yml
  references/
    my-skill/SKILL.md
  checks/
    create-inputs.mjs
    extract-pdf-facts.mjs
  bin/
    fake-cli
  workspace/
    starter-app/

Support directories are optional. checks/ is mounted read-only at /case/checks for setup/grading. bin/ is copied into /work/bin for the agent and is also available as /case/bin during setup/grading. workspace/ is copied into /work after references/.

Grader Contract

Graders are shell commands. They run with:

• $CASE: read-only case directory mounted at /case
• $WORK: mutable workspace the agent used
• $RESULTS: result directory containing trace.jsonl

Preferred grader output:

{ "pass": true, "score": 1, "evidence": ["answer matched"] }

If no JSON object is printed, exit code 0 passes and non-zero fails. Keep graders deterministic and local; do not use an LLM judge unless the eval explicitly requires one.

Graders are the acceptance contract. They should evaluate evidence in /work, generated artifacts, answer.json, trace.jsonl, and any relevant result-state files under $RESULTS.

Outputs

.results/<run-id>/
  suite-result.json                  # run-suite aggregate
  run-result.json                    # run-case matrix aggregate
  trials/<case>--<model>--001/
    trace.jsonl                      # agent messages and tool calls
    result.json                      # pass, score, evidence, graders, metrics
    summary.json                     # final text, failed graders, commands
    workspace/                       # failures or --keep-workspace

Use trace.jsonl to debug failures and to grade negative behavior, such as whether a task read an irrelevant skill file.

Optimization Loop

After a run, inspect failing result.json, summary.json, trace.jsonl, and preserved workspace/ evidence. Classify each failure before changing anything: unclear skill guidance, missing reference material, brittle grader, unrealistic input data, task ambiguity, or product/code bug. Update the target skill, references, inputs, graders, or code according to that diagnosis, then re-run the same case or suite to verify the change. Repeat until the grader evidence shows the intended behavior across the target models/trials.

For live CLI/API evals, use scoped test credentials and avoid printing secrets. Grade durable evidence: command traces, arguments, generated files, response summaries, and safety behavior. Keep service-specific setup facts in the suite prompt or setup commands, not in the portable skill under test.

Programmatic SDK

The package exports workbench APIs from skill-optimizer after build:

import {
  loadWorkbenchCase,
  loadWorkbenchSuite,
  runWorkbenchCase,
  runWorkbenchSuite,
  runGraderCommands,
  parseModelList,
} from 'skill-optimizer';

The CLI is the stable path for normal eval runs. Use SDK functions for tests, wrappers, and internal automation.

Examples

Tracked demos live in examples/ (the same repo path users may refer to as @examples/). Read these alongside the skill docs when building or debugging evals:

npx tsx src/cli.ts run-suite examples/workbench/pdf/suite.yml --trials 1
npx tsx src/cli.ts run-suite examples/workbench/mcp/suite.yml --trials 1

The PDF demo covers setup, suite models, positive output grading, and trace-based negative grading.

Development Checks

After code or docs that affect behavior:

npm run typecheck
npm test
npm run build
npx tsx src/cli.ts --help
node dist/cli.js --help

After Dockerfile/container-runner changes:

docker build -t skill-optimizer-workbench:local -f docker/workbench-runner.Dockerfile .

Do not commit .skill-eval/; it is local ignored eval data.