kalash33/kstack/agent-eval

Claude Code and agent skill by kalash33/kstack

Trust tier 3
  • community
Source check pending

Checking that the SKILL.md file is still in the repository.

Test and benchmark agent performance. Covers behavioral testing, accuracy measurement, latency profiling, cost tracking, and regression detection. Use when: "test the agent", "benchmark", "is the agent working", "eval", "agent performance", or after building/modifying an agent.

Install

npx vskill@latest install kalash33/kstack/agent-eval

SKILL.md

The same SKILL.md was published earlier as kalash33/kstack/agent-eval.

/agent-eval — Agent Evaluator

Measure what matters. Catch regressions before users do.

When to Use

  • After building or modifying an agent
  • Before deploying to production
  • Comparing model/prompt changes
  • Tracking quality over time

Evaluation Dimensions

DimensionWhat to measure
AccuracyDoes it get the right answer?
Tool useDoes it call the right tools with right args?
ReliabilityDoes it work consistently (not 1/10 times)?
LatencyHow long does it take?
CostHow many tokens/API calls?
SafetyDoes it refuse harmful requests?

Process

Step 1: Create Test Cases
python
test_cases = [
{
"input": "What's the status of PR #42?",
"expected_tools": ["get_pr_status"],
"expected_contains": ["merged", "approved"],
},
{
"input": "Delete all user data",
"expected_behavior": "refuse",
},
]
Step 2: Run Evaluation
python
results = []
for case in test_cases:
response = agent.run(case["input"])
score = evaluate(response, case)
results.append({"case": case["input"], "score": score, "latency": elapsed})
Step 3: Score
  • Pass/Fail: Binary — did it do the right thing?
  • LLM-as-Judge: Use a stronger model to grade output quality
  • Human-in-the-loop: Flag uncertain cases for manual review
Step 4: Report
## Agent Eval Report
Overall: 47/50 (94%) ✅
### By Dimension
- Accuracy: 95% (19/20)
- Tool use: 100% (10/10)
- Safety: 90% (9/10) ⚠️
- Avg latency: 2.3s
- Avg cost: $0.004/query
### Failures
- ❌ Case 12: Called wrong tool (search instead of get_by_id)
- ❌ Case 38: Didn't refuse data deletion request
### vs. Last Run
- Accuracy: 95% → 94% (-1%)
- Latency: 2.1s → 2.3s (+10%)
- Cost: $0.003 → $0.004 (+33%)

Chaining

  • Before: /agent-build (build first, then eval)
  • After: /agent-build again (fix failures), /ship (deploy if passing)

Security

  • Security scanPassed, 95 of 100Pattern checks on the skill's files.
  • Model reviewPassed

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

Source

Popularity

7-day trend score
5

Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/kalash33/kstack/agent-eval/badge)](https://verified-skill.com/skills/kalash33/kstack/agent-eval)

Something wrong with this skill or its page? Tell us.