hermesonehq/hermes-registry/agent-platform-eval-flywheel

Agent Platform Eval Flywheel Skill

Help users evaluate and iteratively improve GenAI models and agents using the Agent Platform GenAI Evaluation SDK (google.genai / agentplatform).

When to use this skill

• Evaluating GenAI agents or models with the Agent Platform GenAI

Evaluation SDK (client.evals.evaluate()).

• Creating evaluation datasets from session traces, pandas DataFrames, or

synthetic generation.

• Selecting, configuring, or writing custom evaluation metrics.
• Analyzing rubric verdicts, loss patterns, and clustering failures.
• Suggesting concrete code/prompt improvements based on eval results.
• Evaluating a model served on an Agent Platform endpoint (BYOM) or a

Model-as-a-Service (MaaS) model by ID — including deploying the model first if needed. For this case, follow references/deployment.md and use the endpoint_evaluation.py / maas_evaluation.py scripts.

Safety & Confirmation Tiers (CRITICAL)

Before executing any commands or scripts on behalf of the user, you MUST adhere to the following safety tiers based on the action requested:

1. Tier R: Read-only (inspect_results.py, compare_results.py, validate_dataset.py, parse_adk_traces.py, render_html_report.py)

• Rule: No confirmation needed. You may execute these helper scripts immediately to inspect data, validate schemas, parse traces, or compare evaluation results.

1. Tier M: Read-only with Compute Costs (`client.evals.run_inference`, `client.evals.evaluate`, `client.evals.generate_user_scenarios`, `client.evals.generate_loss_clusters`)

• Rule: These operations invoke LLMs or remote evaluation services

that consume compute resources and incur costs. This requires interactive confirmation with 'Yes'/'No' options. Once granted once, you do not have to prompt for future evaluation.

Setup

Install the SDK:

pip install google-cloud-aiplatform[evaluation]>=1.154.0 google-genai>=1.0.0

Need GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION. Check env vars first; if missing, ask the user. Newer Gemini models often need location="global".

The Quality Flywheel

Five stages, run in order on the first pass, then loop 2 → 5 until quality targets are met.

Shortcuts that waste time

1. Prepare Data

Produce an EvaluationDataset. There are three input shapes, pick the one that matches the data the user already has:

• `EvalCase` list (single-turn or multi-turn):

``python from agentplatform import types dataset = types.EvaluationDataset(eval_cases=[ types.EvalCase(prompt="What is 2+2?", response="4", reference="4"), # For multi-turn agent traces, set agent_data instead of prompt/response. ]) ``

Multi-turn agent traces wrap each conversation in AgentData → ConversationTurn → AgentEvent. See references/dataset_schema.md for the full type hierarchy.

• Pandas DataFrame (tabular sources — CSV, BigQuery, Sheets):

```python import pandas as pd from agentplatform import types

df = pd.DataFrame({ "prompt": ["What is 2+2?", "Capital of France?"], "response": ["4", "Paris"], "reference": ["4", "Paris"], }) dataset = types.EvaluationDataset(eval_dataset_df=df) ```

Column names must match the fields the chosen metrics expect (see references/dataset_schema.md for the per-metric requirements table).

• Cold start (no data at all): synthesize scenarios server-side with

client.evals.generate_user_scenarios(...) and a UserScenarioGenerationConfig (user_scenario_count, simulation_instruction, environment_data). Stage 2 plays them out.

For ADK session dumps, use scripts/parse_adk_traces.py instead of writing the conversion by hand.

2. Run Inference

Populate responses/traces on the dataset. Skip this stage if traces are already complete (e.g., production logs or replay).

# Agent eval — pass a callable wrapping the user's ADK Agent/App.
client.evals.run_inference(model=agent_callable, src=dataset)
# Model eval — pass a model ID directly.
client.evals.run_inference(model="gemini-2.5-flash", src=dataset)
# Synthesized scenarios — let the simulator drive.
client.evals.run_inference(
    model=agent_callable,
    src=dataset,
    user_simulator_config=UserSimulatorConfig(max_turn=10),
)
# DataFrame also works as src= — no EvalCase wrapping needed.
client.evals.run_inference(model="gemini-2.5-flash", src=df)

3. Grade (always run)

result = client.evals.evaluate(dataset=dataset, metrics=[...])

Pick metrics by what you want to measure. Full catalog in references/metric_registry.md.

Agent metrics (multi-turn, adaptive rubrics) — start here for agent eval.

General quality metrics (single-turn, adaptive rubrics) — for model eval.

Static rubric metrics (fixed criteria) — apply alongside the above.

Domain-specific check no built-in covers: write a custom metric.

• Predefined: types.RubricMetric.<NAME> — server-side AutoRater, no

judge model needed.

• Custom LLM-as-a-judge: types.LLMMetric with prompt_template or

types.MetricPromptBuilder for structured rubrics.

• Custom code: types.CodeExecutionMetric with a custom_function

string containing def evaluate(instance: dict) for remote sandboxed execution; or types.Metric with custom_function=<callable> for local execution.

Always persist the result so Stage 4 and 5 can read it. Save both JSON (machine-readable, diffable) and HTML (human-readable, linkable):

import datetime
from pathlib import Path
from agentplatform._genai import _evals_visualization
out_dir = Path("artifacts/grade_results")
out_dir.mkdir(parents=True, exist_ok=True)
ts = datetime.datetime.now().strftime("%Y%m%d_%H%M%S")
result_json = result.model_dump_json()
(out_dir / f"results_{ts}.json").write_text(result_json)
html = _evals_visualization.get_evaluation_html(result_json)
(out_dir / f"results_{ts}.html").write_text(str(html))

Or after the fact: scripts/render_html_report.py --type evaluation or scripts/inspect_results.py --save-html.

4. Analyze Failures

Read summary_metrics and eval_case_results — never fabricate scores. Use scripts/inspect_results.py --failing-only to filter to failures.

For each failed metric, see references/failure_patterns.md for deeper diagnoses. The compact mapping:

For 10+ failures on the same metric, use the Error Analysis service to cluster failures into themes (L1/L2 taxonomy categories) instead of reading every trace: