mhylle/claude-skills-collection/eval-harness

Eval Harness

<!-- contents: generated by tests/skill_structure.py --write; edit headings, not this block -->

Contents

• Modes
• Evaluation-driven development (EDD)
• Eval types (summary)
• Grader types (summary)
• Metrics (summary)
• Mode: create
• Mode: run
• Mode: report
• Quick reference
• Grader selection guide
• Metric selection guide
• Common thresholds
• References (loaded on demand)

Reference files (open one when a step points to it):

• references/eval-types.md: Eval types
• references/file-format.md: Eval file format
• references/graders.md: Graders
• references/metrics.md: Metrics — pass@k, pass^k, latency, cost

<!-- /contents -->

Build and run evaluation suites for AI features in the user's application — a prompt, an LLM call, a RAG pipeline, an agent. Covers two eval types (capability, regression), three grader families (code, model, human), and the metrics that summarise them (pass@k, pass^k, latency, cost).

Not this skill:

• Evaluating or benchmarking a Claude Code skill or its triggering → the skill-creator skill.
• A Claude API feature where the user wants a full eval set with measured cost, a report viewer, and iterative prompt tuning → offer the claude-api skill's build-eval and hillclimb subcommands, which go further than this skill. If claude-api is not in your skill list, carry on here.

Modes

Without an argument, infer the mode: no suite yet → create; a suite but no results, or the code changed since the last run → run; results present and the user asks how it went → report.

Evaluation-driven development (EDD)

Evals are defined before or alongside implementation, not after, so "working" is explicit and testable from day one.

1. Define success first — write the evals that say what "working correctly" means before building.
2. Measure continuously — run evals during development, not just at the end.
3. Automate where possible — code graders for anything codifiable; model graders for semantic quality; human graders for nuance and for calibrating the model graders.
4. Track regressions — every capability added becomes a regression eval.
5. Don't modify evals to pass — unless the requirements changed, and then update baselines explicitly.

Eval types (summary)

Full structures and YAML examples → references/eval-types.md.

Grader types (summary)

Configs, implementation sketches, judge call and rubric templates → references/graders.md.

Metrics (summary)

Formulas, what to record per case, and the picking guide → references/metrics.md.

Mode: create

Write the suite file and the eval files it references. Eval file format → references/file-format.md.

# evals/eval-suite.yaml
eval_suite:
  name: "Feature X evaluation suite"
  version: "1.0.0"
  description: "Capability and regression evals for Feature X"
 
  target:                                         # how the runner calls the system under test
    entry_point: "src/feature_x/answer.py:answer"  # the real code path, not a rebuilt API call
 
  config:
    reps: 3                     # samples per case; the k available to pass@k and pass^k
    max_workers: 4
    timeout_per_case_seconds: 120
 
  thresholds:
    pass_at_1: 0.85
    pass_hat_3: 0.70
    latency_p95_seconds: 8
 
  evals:
    - $ref: "./capability/cap-001.yaml"
    - $ref: "./capability/cap-002.yaml"
    - $ref: "./regression/reg-001.yaml"
 
  output_dir: "./results"

Checklist:

• Success criteria written as concrete, checkable statements
• Capability evals for the new behaviour, regression evals for what must not break
• A grader per case, code wherever the output can be checked programmatically
• For model graders, the judge model agreed with the user (references/graders.md)
• Thresholds that are realistic for the stage (see Common thresholds below)
• Edge cases (empty input, very long input, adversarial input) covered

While implementing the feature, run the relevant subset often (run with a filter) and use failures to guide the work.

Mode: run

There is no eval-harness CLI. The runner is a small script in the user's project, written in the project's language and idiom, that you generate once and reuse.

1. Look for an existing runner (evals/run_evals.*, an eval script in package.json or the Makefile, a pytest marker). Reuse it if it meets the contract below; extend it rather than writing a second one.
2. Otherwise generate evals/run_evals.<ext> — a pytest module or Click CLI for Python, an npm script for Node, and so on.
3. Run it on two or three cases first and read one written row in full. Fix the runner until every field is present and plausible, then run the whole suite.

Runner contract:

• Input: the suite file path; optional --filter <category-or-tag>, --reps <k>, --run-id <id>.
• Load: parse the suite (YAML or JSON), resolve each $ref, and validate that every test case has input, grader, and expected where its grader needs one.
• Execute: for each case × rep, call the system under test through target, mocking only external side effects (emails, payments, production writes), with the per-case timeout. Record the output, latency_s, and — when the call surfaces them — model and token usage from the response.
• Grade: code graders in-process; model graders call the judge with a structured-output schema and record judge_model and judge_usage; human-graded cases go to review.jsonl with status pending. Details → references/graders.md.
• Write: append one JSON line per (case, rep) to <output_dir>/<run-id>/results.jsonl as each finishes, with case_id, rep, category, tags, status (ok | error | timeout | pending), passed, score, details, latency_s, model, usage, judge_model, judge_usage. Infrastructure failures get status: error, not passed: false, so they aren't counted as model failures. On restart, skip (case, rep) pairs already written.
• Finish: write summary.json (pass rate per category and per grader type, pass@k / pass^k, latency and token summaries per references/metrics.md), print one PASS/FAIL line per threshold, and exit non-zero if any threshold fails, so the run can gate a merge.

Mode: report

Read results.jsonl (plus any grades filled into review.jsonl), compute the metrics in references/metrics.md, and write report.md from a summary like this:

eval_report:
  metadata:
    suite_name: "Feature X evaluation suite"
    run_id: "eval-20261006-143052"
    timestamp: "2026-10-06T14:30:52Z"
    duration_seconds: 127
    environment:
      model: "claude-opus-5-5"      # model under test, as reported by the responses
      app_version: "1.2.3"
 
  summary:
    total_cases: 25
    passed: 22
    failed: 3
    errors: 0
    pending_human: 0
    pass_rate: 0.88
 
    by_category:
      capability:   { total: 15, passed: 13, pass_rate: 0.867 }
      regression:   { total: 10, passed: 9,  pass_rate: 0.90 }
 
    by_grader:
      code:  { total: 18, passed: 17 }
      model: { total: 5,  passed: 4 }
      human: { total: 2,  passed: 1 }
 
  metrics:
    pass_at_1: 0.88
    pass_at_3: 0.96
    pass_hat_3: 0.68
    latency_s: { p50: 2.1, p95: 4.8, max: 7.2 }
    tokens_per_case: { input_median: 1850, output_median: 420 }
    cost_usd: { per_case_median: 0.016, run_total: 1.21, judge_total: 0.41 }   # derived from usage at report time
 
  thresholds:
    - { metric: pass_at_1, value: 0.88, threshold: 0.85, status: PASS }
    - { metric: pass_hat_3, value: 0.68, threshold: 0.70, status: FAIL }
 
  failures:
    - eval_id: "cap-003"
      name: "Complex nested extraction"
      reason: "Missed nested array handling"
    - eval_id: "cap-011"
      name: "Edge case: empty input"
      reason: "Threw exception instead of returning empty"
    - eval_id: "reg-007"
      name: "Code review comment quality"
      reason: "Specificity score 3.6, baseline 4.2 (tolerance -0.3)"
 
  regressions_since_previous_run:
    - "reg-007: passed in eval-20261005-091200, fails now"
 
  recommendations:
    - "Improve nested data structure handling"
    - "Add empty input validation"

Compare with the previous run in the same output_dir when there is one, and list the cases that flipped from pass to fail.

Evals can be run as a manual gate before marking an implementation phase complete: run the suite and check the threshold lines.

Quick reference

Grader selection guide

Metric selection guide

Common thresholds

References (loaded on demand)

• references/eval-types.md — capability and regression eval structures with YAML examples
• references/graders.md — code / model / human grader configs, the structured-output judge call, rubric examples, calibration
• references/metrics.md — pass@k and pass^k formulas, latency and cost recording and summaries, picking guide
• references/file-format.md — eval file schemas (YAML/JSON) and the recommended directory layout