itseffi/ai-product-evals/evaluate-tool-use
Claude Code and agent skill by itseffi/ai-product-evals
- community
Checking that the SKILL.md file is still in the repository.
Guides evaluation of tool-use behavior in agentic systems. Use when measuring tool selection accuracy, identifying wrong-tool or no-tool failures, improving argument checks, or designing stronger tool-use eval cases.
Install
npx vskill@latest install itseffi/ai-product-evals/evaluate-tool-useSKILL.md
The same SKILL.md was published earlier as itseffi/ai-product-evals/evaluate-tool-use.
Evaluate Tool Use
Overview
- Read traces first to identify whether failures are tool-choice errors, argument errors, or parser errors.
- Start from
evals/agent-tools.json. - Evaluate tool choice separately from argument correctness.
- Include explicit no-tool cases so models are not rewarded for unnecessary tool use.
- Tighten the parser only after reviewing how multiple providers format tool calls.
Prerequisites
Inspect evals/agent-tools.json, evals/quick-test.json, evaluators/index.mjs, and traces/ before changing the tool-use suite. Determine whether failures come from ambiguous prompts, weak argument expectations, or parser brittleness.
Core Instructions
Separate Tool Choice From Argument Correctness
Measure:
- whether the correct tool was chosen
- whether no tool should have been chosen
- whether the arguments are semantically correct
- whether formatting broke parsing even when intent was correct
Do not treat all tool failures as one category.
Build Better Tool-Use Coverage
Add cases for:
- obvious tool usage
- ambiguous tool usage
- explicit no-tool usage
- multi-argument tools
- distractor tools with overlapping semantics
The suite should measure both over-calling and under-calling tools.
Make Arguments Realistic
Expected arguments should reflect what downstream code actually needs. If argument matching is too loose, broken tool calls can pass. If argument matching is too strict, reasonable provider formatting differences can fail incorrectly.
Review Parser Behavior Across Providers
Inspect traces to see whether providers produce:
TOOL: name(args)- JSON-like tool calls
- extra formatting around the tool call
- casing or quoting differences
Parser changes should be based on observed outputs, not assumptions.
Repo Files To Inspect
evals/agent-tools.jsonevals/quick-test.jsonevaluators/index.mjsrun-eval.mjstraces/
Anti-Patterns
- Evaluating only obvious tool calls.
- Ignoring no-tool cases.
- Treating partial argument overlap as full correctness.
- Tightening the parser without reading traces from multiple providers.
- Assuming formatting errors and reasoning errors are the same problem.
Security
- Security scanPassed, 95 of 100Pattern checks on the skill's files.
- Model reviewPassed
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingskills/evaluate-tool-use/SKILL.md
Popularity
- Repository stars
- 3
- 7-day trend score
- 12
Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/itseffi/ai-product-evals/evaluate-tool-use)