kienbui1995/magic-powers/agentic-eval
Claude Code and agent skill by kienbui1995/magic-powers
- community
Checking that the SKILL.md file is still in the repository.
Use when evaluating AI agent systems — trajectory evaluation, pass@k testing, tool call correctness, non-deterministic behavior testing, and building eval infrastructure specific to multi-step agentic workflows.
Install
npx vskill@latest install kienbui1995/magic-powers/agentic-evalSKILL.md
The same SKILL.md was published earlier as kienbui1995/magic-powers/agentic-eval.
Agentic Eval
Overview
Evaluating an agent is fundamentally different from evaluating a single LLM call. The unit of evaluation is a trajectory — the full sequence of plan, tool calls, observations, and final response. A correct final answer via a bad trajectory is still a bad agent.
When to Use
- Evaluating a multi-step AI agent (not just a single LLM call)
- Testing tool use correctness and sequence quality
- Handling non-determinism in agent behavior
- Setting up CI/CD eval for an agent system
- Diagnosing WHY an agent fails, not just THAT it fails
Core Jobs
1. Why Agentic Eval ≠ LLM Eval
| Dimension | LLM Eval | Agentic Eval | |
|---|---|---|---|
| Unit | 1 prompt → 1 response | Multi-step: plan → tool calls → synthesis | |
| Evaluation | Compare output to expected | Evaluate trajectory + output | |
| Failure modes | Hallucination, off-topic | Wrong tool, wrong params, infinite loop, tool hallucination | |
| Non-determinism | Minor variation | Compound across steps — run multiple times | |
| Test design | Golden dataset | Scenario-based with sandbox tools |
Key implication: an agent can produce the correct final answer via a completely wrong path (e.g., lucky guess after repeated failures). Trajectory evaluation catches this; output-only evaluation does not.
2. Three-Layer Trajectory Evaluation
Layer 1: Final response evaluation ("Did it work?")
Use this as a quick pass/fail gate before deeper analysis. Necessary but not sufficient.
Layer 2: Trajectory evaluation ("Where did it go wrong?")
Prefer any_order or subset mode — valid agents often find multiple correct paths. Reserve exact mode for security-critical flows where order matters.
Layer 3: Step-level evaluation ("Why did it fail?")
Step-level breakdown exposes whether failures are systemic (always wrong tool) or situational (wrong params in edge cases).
3. Tool Hallucination Detection
Tool hallucination = agent describes calling a tool in its reasoning but never actually invokes it. This is always high severity — the agent is lying about its actions.
CI rule: tool hallucination rate must be 0.0. Any hallucination is a blocker. An agent that fabricates its own tool calls cannot be trusted in production.
4. Pass@k for Non-Deterministic Agents
Agents are non-deterministic. A single test run that passes or fails tells you little. Pass@k measures the probability that at least 1 of k sampled runs succeeds.
Target thresholds by use case:
| Use case | Recommended target | |
|---|---|---|
| Production customer-facing agent | pass@1 > 0.85 | |
| Internal tooling agent | pass@3 > 0.90 | |
| Research prototype | pass@5 > 0.70 |
Always report pass@1 AND pass@3 — an agent with pass@1=0.40 and pass@3=0.95 is fine for a retry-based system but unacceptable for single-shot production flows.
5. Sandbox Tool Environment
Test agents with mocked tools to avoid hitting real APIs — keeps eval fast, deterministic, and cheap.
Scenarios define what mock tools return for each test situation. This makes tests reproducible — same input always produces same tool responses, exposing agent logic flaws rather than API flakiness.
6. Scenario-Based Test Design
Agent tests are scenarios, not golden examples. Each scenario defines: initial context, available mock tools, expected trajectory pattern, and pass criteria.
7. Agent-Specific CI Pipeline
CI blocks on: any tool hallucination, pass@1 below threshold, trajectory precision below 0.75. CI warns on: pass@1 between threshold and 0.90 (approaching degradation zone).
Key Concepts
- Trajectory — the complete sequence of steps (reasoning, tool calls, observations, synthesis) an agent takes to complete a task; the primary evaluation unit in agentic eval
- Pass@k — probability that at least 1 of k sampled runs of the agent succeeds; the correct metric for non-deterministic systems
- Tool hallucination — agent describes calling a tool in its reasoning without actual invocation; always high severity, zero-tolerance in production
- Step precision — what fraction of actual tool calls were necessary (unnecessary calls = inefficiency or confusion)
- Step recall — what fraction of necessary tool calls were actually made (missing calls = incomplete work)
- Trajectory precision/recall — step precision and recall aggregated across all steps in a run
- Sandbox environment — mocked tool execution environment for reproducible, API-free testing
- Scenario — a test case for agents: initial context + mock tool responses + expected trajectory + pass criteria
Checklist
- [ ] Using pass@k (not single run) for all agent test cases?
- [ ] Sandbox environment set up to mock all tool calls?
- [ ] Tool hallucination detection enabled in eval pipeline?
- [ ] Three-layer eval implemented: final response + trajectory + step-level?
- [ ] CI blocks on tool hallucination rate > 0% (zero tolerance)?
- [ ] Trajectory eval uses
any_orderorsubsetmode (not rigid exact match) for flexible agents? - [ ] Test scenarios cover failure cases: tool errors, timeouts, ambiguous input, fraud signals?
- [ ] Pass@1 target documented per use case (production vs internal vs prototype)?
- [ ] Eval results tracked over time to detect regression trends?
- [ ] Scenario library covers at least happy path + 2 edge cases per agent capability?
Key Outputs
- Pass@k scores by scenario (pass@1 and pass@3 minimum) with trend vs baseline
- Tool hallucination report — zero tolerance, any hallucination is a release blocker
- Trajectory precision/recall by scenario, broken down by step
- Step-level failure breakdown — wrong tool selected vs wrong parameters vs unsound reasoning
- CI pass/fail status with failure explanation pointing to specific scenarios and layers
- Scenario library — versioned test cases with mock tool responses for reproducibility
Output Format
This is the start of the file. Read the full SKILL.md on GitHub.
Security
- Security scanPassed, 100 of 100Pattern checks on the skill's files.
- Model reviewNot runA model review is scheduled for the next rescan.
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingskills/agentic-eval/SKILL.md
Popularity
- 7-day trend score
- 0
Not refreshed yet.
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/kienbui1995/magic-powers/agentic-eval)