watcher-hermes/hermes-skills/benchmark-harness

Claude Code and agent skill by watcher-hermes/hermes-skills

Trust tier 3
  • community
Source check pending

Checking that the SKILL.md file is still in the repository.

Build a SWE-bench-style harness for a codebase with FAIL_TO_PASS / PASS_TO_PASS gating, contamination checks, and step-count metrics.

Install

npx vskill@latest install watcher-hermes/hermes-skills/benchmark-harness

SKILL.md

The same SKILL.md was published earlier as watcher-hermes/hermes-skills/benchmark-harness.

Given a codebase and a list of (bug, fix) pairs, build a benchmark harness that gates on real unit tests and records operational metrics.

Produce:

  1. Per-task definition: (tid, description, state_before, fail_to_pass_tests, pass_to_pass_tests, solution).
  2. A runner that applies the agent's patch, runs the repo's test suite in a sandbox, and records: FTP pass count, PTP pass count, step count, tokens, wall-clock, cost.
  3. A contamination check: pattern-match the issue text against the produced patch; flag >=30% overlap.
  4. A reporter that emits per-task and aggregate scores as JSON, plus P50/P75/P95 step and cost.
  5. A CI job that runs the harness on every PR and fails on >=5% regression.

Hard rejects:

  • Harness that reports only a single aggregate number. Require per-task results + distributions.
  • Harness that runs tests without a sandbox. Agent-provided patches are untrusted code.
  • Harness with no PASS_TO_PASS gate. Patches that break other tests silently regress the product.

Refusal rules:

  • If the user asks for "just the FAIL_TO_PASS score," refuse. Add PASS_TO_PASS; breaking existing tests is a worse regression than missing the fix.
  • If the tests are not pinned to a specific commit, refuse. Drift in tests makes scores incomparable across runs.
  • If the tasks overlap with issue text seen during training, flag it explicitly.

Output: tasks.py, harness.py, contamination.py, report.py, README.md explaining the sandbox, the gates, the contamination policy. End with "what to read next" pointing to Lesson 30 for eval-driven development on top of the harness.

Security

  • Security scanPassed, 95 of 100Pattern checks on the skill's files.
  • Model reviewPassed

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

Source

Popularity

7-day trend score
5

Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/watcher-hermes/hermes-skills/benchmark-harness/badge)](https://verified-skill.com/skills/watcher-hermes/hermes-skills/benchmark-harness)

Something wrong with this skill or its page? Tell us.