itseffi/ai-product-evals/evaluate-tool-use

Claude Code and agent skill by itseffi/ai-product-evals

Trust tier 3
  • community
Source check pending

Checking that the SKILL.md file is still in the repository.

Guides evaluation of tool-use behavior in agentic systems. Use when measuring tool selection accuracy, identifying wrong-tool or no-tool failures, improving argument checks, or designing stronger tool-use eval cases.

Install

npx vskill@latest install itseffi/ai-product-evals/evaluate-tool-use

SKILL.md

The same SKILL.md was published earlier as itseffi/ai-product-evals/evaluate-tool-use.

Evaluate Tool Use

Overview

  1. Read traces first to identify whether failures are tool-choice errors, argument errors, or parser errors.
  2. Start from evals/agent-tools.json.
  3. Evaluate tool choice separately from argument correctness.
  4. Include explicit no-tool cases so models are not rewarded for unnecessary tool use.
  5. Tighten the parser only after reviewing how multiple providers format tool calls.

Prerequisites

Inspect evals/agent-tools.json, evals/quick-test.json, evaluators/index.mjs, and traces/ before changing the tool-use suite. Determine whether failures come from ambiguous prompts, weak argument expectations, or parser brittleness.

Core Instructions

Separate Tool Choice From Argument Correctness

Measure:

  • whether the correct tool was chosen
  • whether no tool should have been chosen
  • whether the arguments are semantically correct
  • whether formatting broke parsing even when intent was correct

Do not treat all tool failures as one category.

Build Better Tool-Use Coverage

Add cases for:

  • obvious tool usage
  • ambiguous tool usage
  • explicit no-tool usage
  • multi-argument tools
  • distractor tools with overlapping semantics

The suite should measure both over-calling and under-calling tools.

Make Arguments Realistic

Expected arguments should reflect what downstream code actually needs. If argument matching is too loose, broken tool calls can pass. If argument matching is too strict, reasonable provider formatting differences can fail incorrectly.

Review Parser Behavior Across Providers

Inspect traces to see whether providers produce:

  • TOOL: name(args)
  • JSON-like tool calls
  • extra formatting around the tool call
  • casing or quoting differences

Parser changes should be based on observed outputs, not assumptions.

Repo Files To Inspect
  • evals/agent-tools.json
  • evals/quick-test.json
  • evaluators/index.mjs
  • run-eval.mjs
  • traces/

Anti-Patterns

  • Evaluating only obvious tool calls.
  • Ignoring no-tool cases.
  • Treating partial argument overlap as full correctness.
  • Tightening the parser without reading traces from multiple providers.
  • Assuming formatting errors and reasoning errors are the same problem.

Security

  • Security scanPassed, 95 of 100Pattern checks on the skill's files.
  • Model reviewPassed

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

Source

Popularity

Repository stars
3
7-day trend score
12

Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/itseffi/ai-product-evals/evaluate-tool-use/badge)](https://verified-skill.com/skills/itseffi/ai-product-evals/evaluate-tool-use)

Something wrong with this skill or its page? Tell us.