shihongdev/evalyn/evalyn-eval

evalyn-eval

Overview

Build a dataset from traces, auto-recommend metrics based on trace analysis, and run evaluation. This skill reads actual trace data to make metric recommendations rather than asking abstract questions.

Pre-flight

1. Verify traces exist:

evalyn list-calls --limit 5

If no traces: "You need to instrument your agent first. Invoke evalyn-setup."

1. Check if a dataset already exists:

ls data/*/dataset.jsonl 2>/dev/null

If dataset exists, skip to Step 2.

Step 1: Build Dataset

Identify the project name from the evalyn list-calls output (project column).

evalyn build-dataset --project <project-name>

Capture the output path - it prints "Wrote N items to <path>". Use this path for all subsequent commands.

Step 2: Auto-Recommend Metrics

Inspect a trace to understand the agent's behavior:

evalyn show-trace --last -v

Analyze the trace structure and recommend a bundle. Evalyn has 17 curated metric bundles:

To see all available bundles:

evalyn suggest-metrics --mode bundle --help

Apply the recommended bundle:

evalyn suggest-metrics --dataset <path> --mode bundle --bundle <recommended>

Then expand coverage with LLM-based selection from the full 130+ metric registry:

evalyn suggest-metrics --dataset <path> --mode llm-registry --append

This two-pass approach gives a solid base (curated bundle) plus tailored additions (LLM picks from full registry).

Available metric modes

Do NOT use modes like agent, rag, or classify - those do not exist.

Step 3: Run Evaluation

evalyn run-eval --dataset <path>

This runs all metrics, generates results.json in eval_runs/, and prints a summary table. Note the run ID from the output.

Useful flags:

• --workers 8: increase parallel workers (default 4, max 16)
• --provider openai: use OpenAI instead of Gemini for LLM judges
• --provider ollama: use local Ollama models

Hand-off

"Evaluation complete. Invoke evalyn-analyze to dig into the results, identify failures, and get recommendations."