googlecloudplatform/professional-services/agent-eval
Claude Code and agent skill by googlecloudplatform/professional-services
- community
Checking that the SKILL.md file is still in the repository.
Executes high-performance agent evaluations, multi-turn UserSim simulations, and declarative metric grading aligned with google/agents-cli and the Quality Flywheel. Publishes benchmark artifacts to the GCS Evaluation Registry, executes automated head-to-head delta comparisons (--compare-to), and optimizes system instructions via ADK GEPA (Genetic Evolutionary Prompt Optimization). Use when running agent benchmarks, evaluating ADK/FastAPI agents, diagnosing loss clusters, comparing prompt iterations, running GEPA prompt optimization, or serving evaluation dashboards. Don't use for raw agent code scaffolding (use google-agents-cli-scaffold) or infrastructure deployment (use google-agents-cli-deploy).
Install
npx vskill@latest install googlecloudplatform/professional-services/agent-evalSKILL.md
The same SKILL.md was published earlier as googlecloudplatform/professional-services/agent-eval.
Agent Evaluation & Continuous Optimization Framework
This skill defines the end-to-end evaluation, benchmarking, automated optimization, and publication workflow using the `agent-eval` CLI Pipeline aligned with `google/agents-cli` and the ADK Quality Flywheel (adk.dev/optimize).
1. Reference Architecture & Deep Guides
| Reference Guide | Contents | |
|---|---|---|
| `references/dataset_schema.md` | Canonical dataset.jsonl schema (single-turn, multi-turn, multi-agent topologies). | |
| `references/metrics_guide.md` | Declarative eval_config.yaml specification across 6 metric kinds. | |
| `references/gepa_optimization.md` | Automated genetic prompt evolution via GEPARootAgentPromptOptimizer. | |
| `references/failure_triage.md` | 2-Tier loss clustering and Context Engineering remediation strategies. |
2. The Standard agent-eval run Benchmark Command
Always execute benchmark sweeps using the standardized --feature, --tag, and --publish taxonomy against an active API server endpoint (--base-url):
Key CLI Flags & Defaults
- `--feature`: Git feature branch or capability under test (defaults to active git branch).
- `--tag`: Concise iteration slug (e.g.
direct-bypass-v1,calibrated-prompt-v2). - `--description`: Human-readable context saved into
eval_summary.jsonand rendered in the dashboard. - `--sim-parallelism 6`: Runs 6 scenarios in parallel, cutting multi-turn sweeps down to ~2.5 minutes.
- `--publish`: Automatically syncs the entire output run directory to Google Cloud Storage (
gs://<PROJECT_ID>-eval-artifacts/runs/<run_id>/). - `--compare-to`: Resolves a baseline run (locally or directly from GCS) and generates delta percentage scorecards.
3. The 5-Stage Quality Flywheel Loop
Stage 1: Prepare Data (dataset.jsonl)
- Ensure test scenarios are defined in
tests/eval/dataset.jsonl(see `references/dataset_schema.md`).
Stage 2 & 3: Generate Traces & Grade
agent-eval runexecutes the end-to-end collect → score → analyze loop.- Or drive individual phases:
- Generate Traces:
agent-eval generate --agent-dir app --sim-parallelism 6(alias:simulate/interact) - Grade Traces:
agent-eval grade --traces tests/eval/results/<run_id>/raw/processed_interaction_sim.jsonl(alias:evaluate)
Stage 4: Compare Runs & Analyze Failures
- Compute head-to-head deltas between candidate and baseline runs:
- Or perform in-depth Gemini root-cause analysis:
Stage 5: Automated Prompt Optimization (ADK GEPA)
- When prompts underperform, run ADK's `GEPARootAgentPromptOptimizer` rather than manual trial-and-error:
4. Central GCS Evaluation Registry & Dashboard Access
All runs published with --publish are indexed in real time by the Evaluation Registry Viewer:
- Local / Network Reverse Proxy:
http://localhost:8550 - Cloud Run Endpoint:
https://<EVAL_VIEWER_SERVICE>.run.app - Direct Deep Link:
http://localhost:8550/?run_id=<RUN_ID> - Direct Standalone HTML Report:
http://localhost:8550/report/<RUN_ID>
5. Critical Operational Gotchas & Guardrails
- Suppress Click Aborts (`AGENT_EVAL_NO_PAUSES=1`): Always export
AGENT_EVAL_NO_PAUSES=1in headless scripts, subagents, and tests to prevent Click prompts from hanging on stdin. - Always Use `--base-url` Instead of `--in-process`: OpenTelemetry global tracers cannot be safely forked across multi-process workers in
--in-processmode. Running against an active--base-urlenables full 6-worker concurrency. - PyOpenSSL Multithreading Locks: Batch LLM grading can trigger
Context has already been used to create a Connection. Ensureurllib3.contrib.pyopenssl.extract_from_urllib3()is called to delegate to Python's nativesslengine. - Clear Background Ports Before Starting Servers: Run
fuser -k 8080/tcp(or8502/tcp) before launching newuvicornoradk api_serverdaemons to avoid[Errno 98] address already in use. - HTML Report Type Safety: When sorting iterations in
_build_iterations_data, ensure_natural_keyreturns homogeneous(float, str)tuples to prevent string-vs-int comparison errors.
Security
- Security scanPassed, 100 of 100Pattern checks on the skill's files.
- Model reviewNot runA model review is scheduled for the next rescan.
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingtools/agent-eval/skills/agent-eval/SKILL.md
Popularity
- Repository stars
- 3,070
- Repository forks
- 1,469
- 7-day trend score
- 40
Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/googlecloudplatform/professional-services/agent-eval)