sickn33/agentic-awesome-skills/ai-agent-evaluation-benchmarking
Claude Code and agent skill by sickn33/agentic-awesome-skills
- community
Checking that the SKILL.md file is still in the repository.
Autonomous AI agent benchmark evaluation register: task completion rates, planning accuracy, tool invocation precision, and cost benchmarks.
Install
npx vskill@latest install sickn33/agentic-awesome-skills/ai-agent-evaluation-benchmarkingSKILL.md
The same SKILL.md was published earlier as sickn33/agentic-awesome-skills/ai-agent-evaluation-benchmarking.
AI Agent Capability Evaluation & Benchmarking
What it is: Standardizes multi-metric capability benchmarking, token cost efficiency, and regression monitoring across autonomous coding agents.
Overview
Provides a standardized, auditable framework and data model for AI Agent Capability Evaluation & Benchmarking operations across distributed engineering and decentralized application systems.
When to Use This Skill
- When recording agent evaluation runs with a fixed dataset, rubric, runtime/model revision and measured outcomes.
- When comparing a candidate with a baseline under matched tasks, environment, tool permissions and cost accounting.
- When generating reproducible CSV, SQL DDL, JSON Schema, and Notion property registers for tracking compliance.
How It Works
- Define the parameters, thresholds, and identity bindings required for the target operational register.
- Select appropriate boundary enforcement values from validated enum select sets.
- Export standardized artifacts (CSV table, SQL DDL, JSON Schema) to integrate into validation CI pipelines.
Field Reference
| # | Field Name | Type | SQL Type | JSON Schema Type | Notion Property Type | Example Value | |
|---|---|---|---|---|---|---|---|
| 1 | Benchmark Run ID | id | SERIAL PRIMARY KEY | integer | Text | 1 | |
| 2 | Evaluated Agent Model | select | VARCHAR(64) | string | Select | Claude 3.7 Sonnet | |
| 3 | Benchmark Suite Domain | select | VARCHAR(64) | string | Select | SWE-bench Verified | |
| 4 | Tasks Evaluated Count | number | INTEGER | integer | Number | 100 | |
| 5 | Pass Rate Percentage | number | NUMERIC(5,2) | number | Number | 78.40 | |
| 6 | Tool Hallucination Rate % | number | NUMERIC(5,2) | number | Number | 0.60 | |
| 7 | Average Tokens Per Task | number | INTEGER | integer | Number | 42500 | |
| 8 | Cost Per Solved Task USD | currency | NUMERIC(8,4) | number | Number | 0.3420 | |
| 9 | Regression Verdict | select | VARCHAR(32) | string | Select | Improved with evidence | |
| 10 | Evaluation Lead | text | VARCHAR(64) | string | Text | Ranjeet2063 | |
| 11 | Benchmark Execution Date | date | DATE | string, format: date | Date | 2026-10-01 |
Select Options
Evaluated Agent Model
Benchmark Suite Domain
Regression Verdict
Measurement procedure
Record an immutable run identifier, suite revision, task IDs, model/runtime/tool settings, completed/failed/skipped counts and actual charges. Derive pass rate from eligible completed outcomes under a declared denominator; tool hallucination needs a defined invalid-call denominator. For a regression verdict, use paired outcomes or a justified test and interval, and retain an inconclusive state. Labels and example values below are synthetic, not measured results. Define any relation field in every emitted format only when the target exists and is confirmed.
Relations
Audit Reference-> links to the formal review documentation or test repository.Target Architecture-> links to the deployed contract or autonomous agent runtime component.
Examples
Prompt
Recommended Next Step
Generate the unified field schema, SQL DDL migration, and JSON validation schema to register into your system catalog.Workflow: Define criteria -> Run automated verification -> Record baseline -> Monitor invariants.
Best Practices
- Enforce strict typing on numerical bounds and currency amounts; avoid unstructured free-text fields for critical states.
- Re-run validation test suites on every state-altering commit or parameter change.
- Keep example data synthetic and isolated from production cryptographic keys or private endpoints.
Limitations
- This register does not execute benchmarks, prove compliance or need on-chain RPC credentials. An actual authorized harness and evaluation evidence are external prerequisites.
- Scores are comparable only with matched suite/revision, eligibility, failure/timeout handling and sampling. Record counts and uncertainty, not merely a percentage.
- Define cost per solved task as total attributable run cost divided by solved tasks; zero solves means unavailable/infinite cost under an explicit policy, not zero. Token estimates alone omit provider/tool charges.
Security & Safety Notes
- Generating a register is distinct from running paid providers, transmitting private tasks, changing CI or deploying; those effects need actual authorization.
- Use synthetic dummy keys and mock addresses in test suites and local verification scripts.
Common Pitfalls
- Problem: Mismatched decimal precision between contract runtime and database register.
Solution: Always verify decimals using the explicit field mapping in this reference.
- Problem: Missing authorization checks prior to state update.
Solution: Cross-validate against the Security Audit register before deployment.
Related Skills
- @ai-agent-tool-routing - covers tool schema registration and retry policy.
- @ai-prompt-regression-testing - covers prompt regression baselines and drift.
- @ai-code-generation-guardrails - covers static guardrails for generated code.
Reusable Prompt
Security
- Security scanPassed, 100 of 100Pattern checks on the skill's files.
- Model reviewNot runA model review is scheduled for the next rescan.
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingskills/ai-agent-evaluation-benchmarking/SKILL.md
Popularity
- Repository stars
- 47,174
- Repository forks
- 6,873
- 7-day trend score
- 54
Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/sickn33/agentic-awesome-skills/ai-agent-evaluation-benchmarking)