googlecloudplatform/professional-services/eval-breakdown

Claude Code and agent skill by googlecloudplatform/professional-services

Trust tier 2
  • community
LLM review pending
Source check pending

Checking that the SKILL.md file is still in the repository.

Performs an exhaustive, question-by-question narrative diagnostic breakdown of an agent-eval benchmark run by analyzing question_answer_log.md, eval_summary.json, and raw trajectory traces. Use when diagnosing low score causes, investigating the Memory Reuse vs. Traceability rubric clash, performing pre-release failure audits, or examining judge reasoning across individual scenarios. Don't use for running the benchmark CLI pipeline itself (use agent-eval) or automated genetic prompt tuning (use google-agents-cli-eval).

Install

npx vskill@latest install googlecloudplatform/professional-services/eval-breakdown

SKILL.md

The same SKILL.md was published earlier as googlecloudplatform/professional-services/eval-breakdown.

Eval Breakdown: Question-by-Question Diagnostic Analysis

This skill guides an agent through conducting a rigorous, evidence-grounded, question-by-question narrative diagnostic audit of an agent-eval benchmark run.


1. Input Sources & Directory Anatomy

An agent-eval results folder (tests/eval/results/{run_id}/ or gs://<bucket>/runs/{run_id}/) contains three primary diagnostic artifacts:

  1. eval_summary.json: Aggregated metrics, per-question score distributions, and AutoRater judge verdicts.
  2. question_answer_log.md: Multi-turn conversational transcripts including user prompts, agent responses, tool calls, and tool outputs.
  3. gemini_analysis.md: Automated executive diagnosis and loss cluster summary.

2. Step-by-Step Diagnostic Workflow

Step 1: Generate the Baseline Matrix Table

Extract individual question scores and judge reasoning using the bundled script:

bash
python3 tools/agent-eval/skills/agent-eval/scripts/parse_eval_summary.py \
--summary-path tests/eval/results/{run_id}/eval_summary.json
Step 2: Perform the Dialogue Audit Protocol

Inspect question_answer_log.md for each scenario, analyzing across:

  1. Turn-by-Turn User Intent: What the user requested in Turn 1, Turn 2, etc.
  2. Tool Execution & Traceability: Did the agent emit SQL/API tool calls on each turn where factual assertions were made, or did it answer from empirical dialogue memory?
  3. The Memory Reuse vs. Traceability Rubric Clash:
  • Diagnostic Pattern: If tool_use_quality drops to 0.00 on Turn 2 follow-ups, verify if the agent answered correctly from dialogue memory without re-querying the backend. If the rubric expects a tool call on every turn, calibrate the rubric or prompt instructions accordingly.
  1. Judge Explanation Grounding: Cross-reference any rubric score $< 1.00$ against the exact model response to verify whether the deduction was a genuine agent failure (Tier 1) or an overly strict judge rubric (Tier 2).

3. Executive Output Template

Conclude the audit with a structured summary table:

Q#Scenario IDMetrics SummaryRoot-Cause Finding
Q1TC-001tool_use=1.0, business_logic=1.0✅ PASS (Strict parameter adherence)
Q2TC-002tool_use=0.0, business_logic=1.0⚠️ Memory reuse on Turn 2; rubric expected re-query

Security

  • Security scanPassed, 100 of 100Pattern checks on the skill's files.
  • Model reviewNot runA model review is scheduled for the next rescan.

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

Source

Popularity

Repository stars
3,070
Repository forks
1,469
7-day trend score
40

Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/googlecloudplatform/professional-services/eval-breakdown/badge)](https://verified-skill.com/skills/googlecloudplatform/professional-services/eval-breakdown)

Something wrong with this skill or its page? Tell us.