shihongdev/evalyn/evalyn-analyze

evalyn-analyze

Overview

Analyze evaluation results progressively: summary, insights, failure clustering, and trend analysis. Interpret findings and recommend next actions based on pass rates.

Pre-flight

Verify evaluation runs exist:

evalyn list-runs --limit 3

If no runs: "You need to run an evaluation first. Invoke evalyn-eval."

Identify the latest run ID from the output.

Step 1: Metric Summary

evalyn analyze --run <run-id>

This shows:

• Per-metric pass rates and average scores
• Key findings (highest/lowest performing metrics)
• Overall health rating (GOOD/MODERATE/POOR)

You can also use short IDs (first 8 characters of run ID).

Step 2: Deep Insights

evalyn insights --run <run-id>

Provides diagnostic and prescriptive analysis:

• Metric correlations (which metrics move together)
• Anomaly detection
• Actionable recommendations

Step 3: Compare and Trend

If multiple runs exist (check evalyn list-runs output):

evalyn compare --run1 <previous-run-id> --run2 <latest-run-id>

For longer history across all runs in a dataset:

evalyn trend --project <project-name>

Note: use --run1 and --run2 flags for compare, not positional arguments.

Step 4: Investigate Failures

If any metric has pass rate below 90%:

evalyn cluster-failures --run-id <run-id>

This clusters failed items by failure reason, revealing patterns (e.g., "all failures involve long inputs" or "failures cluster around a specific topic").

Step 5: Interpret and Recommend

Based on the results, recommend next action:

Key distinction to communicate: low pass rates can mean either (a) the agent is actually performing poorly, or (b) the LLM judges are too strict or misaligned. Calibration determines which.

Export

Offer a shareable report:

evalyn export --run <run-id> --format html

Other formats: json, csv, markdown.

Hand-off

If calibration recommended: "Invoke evalyn-calibrate to annotate results and align LLM judges with your expectations."

If agent is performing well: "Your agent looks good. To strengthen confidence, run it on more varied inputs and re-evaluate, or use evalyn simulate --dataset <path> --modes similar,outlier to generate edge cases."