googlecloudplatform/professional-services/eval-breakdown
Claude Code and agent skill by googlecloudplatform/professional-services
- community
Checking that the SKILL.md file is still in the repository.
Performs an exhaustive, question-by-question narrative diagnostic breakdown of an agent-eval benchmark run by analyzing question_answer_log.md, eval_summary.json, and raw trajectory traces. Use when diagnosing low score causes, investigating the Memory Reuse vs. Traceability rubric clash, performing pre-release failure audits, or examining judge reasoning across individual scenarios. Don't use for running the benchmark CLI pipeline itself (use agent-eval) or automated genetic prompt tuning (use google-agents-cli-eval).
Install
npx vskill@latest install googlecloudplatform/professional-services/eval-breakdownSKILL.md
The same SKILL.md was published earlier as googlecloudplatform/professional-services/eval-breakdown.
Eval Breakdown: Question-by-Question Diagnostic Analysis
This skill guides an agent through conducting a rigorous, evidence-grounded, question-by-question narrative diagnostic audit of an agent-eval benchmark run.
1. Input Sources & Directory Anatomy
An agent-eval results folder (tests/eval/results/{run_id}/ or gs://<bucket>/runs/{run_id}/) contains three primary diagnostic artifacts:
eval_summary.json: Aggregated metrics, per-question score distributions, and AutoRater judge verdicts.question_answer_log.md: Multi-turn conversational transcripts including user prompts, agent responses, tool calls, and tool outputs.gemini_analysis.md: Automated executive diagnosis and loss cluster summary.
2. Step-by-Step Diagnostic Workflow
Step 1: Generate the Baseline Matrix Table
Extract individual question scores and judge reasoning using the bundled script:
Step 2: Perform the Dialogue Audit Protocol
Inspect question_answer_log.md for each scenario, analyzing across:
- Turn-by-Turn User Intent: What the user requested in Turn 1, Turn 2, etc.
- Tool Execution & Traceability: Did the agent emit SQL/API tool calls on each turn where factual assertions were made, or did it answer from empirical dialogue memory?
- The Memory Reuse vs. Traceability Rubric Clash:
- Diagnostic Pattern: If
tool_use_qualitydrops to0.00on Turn 2 follow-ups, verify if the agent answered correctly from dialogue memory without re-querying the backend. If the rubric expects a tool call on every turn, calibrate the rubric or prompt instructions accordingly.
- Judge Explanation Grounding: Cross-reference any rubric score $< 1.00$ against the exact model response to verify whether the deduction was a genuine agent failure (Tier 1) or an overly strict judge rubric (Tier 2).
3. Executive Output Template
Conclude the audit with a structured summary table:
| Q# | Scenario ID | Metrics Summary | Root-Cause Finding | |
|---|---|---|---|---|
Q1 | TC-001 | tool_use=1.0, business_logic=1.0 | ✅ PASS (Strict parameter adherence) | |
Q2 | TC-002 | tool_use=0.0, business_logic=1.0 | ⚠️ Memory reuse on Turn 2; rubric expected re-query |
Security
- Security scanPassed, 100 of 100Pattern checks on the skill's files.
- Model reviewNot runA model review is scheduled for the next rescan.
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingtools/agent-eval/skills/eval-breakdown/SKILL.md
Popularity
- Repository stars
- 3,070
- Repository forks
- 1,469
- 7-day trend score
- 40
Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/googlecloudplatform/professional-services/eval-breakdown)