shihongdev/evalyn/evalyn-calibrate

Claude Code and agent skill by shihongdev/evalyn

Trust tier 2
  • community
LLM review pending
Source check pending

Checking that the SKILL.md file is still in the repository.

Use when LLM judges need calibration, evaluation metrics seem misaligned with expectations, or annotation and judge tuning is needed

Install

npx vskill@latest install shihongdev/evalyn/evalyn-calibrate

SKILL.md

evalyn-calibrate

Pre-flight

  1. Verify evaluation runs exist:
bash
evalyn list-runs --limit 1

If no runs: "You need evaluation results first. Invoke evalyn-eval."

  1. Identify calibration targets:
bash
evalyn analyze --run <latest-run-id>

Look for subjective metrics (type [llm]) with pass rates below 90%. Only subjective metrics can be calibrated - objective metrics are deterministic.

Step 1: Annotate Results

Run interactive annotation with per-metric mode for targeted feedback:

bash
evalyn annotate --run-id <latest-run-id> --dataset <dataset-path> --per-metric

This is an interactive terminal session. The user will:

  • See each item's input, output, and LLM judge results
  • Agree or disagree with each metric's judgment
  • Commands: [y]es/pass, [n]o/fail, [s]kip, [v]iew full, [q]uit

Aim for 20-30+ annotations. Focus on disagreements. Annotations save immediately - quit and resume anytime.

Step 2: Run Calibration

Start with the basic optimizer (fast, single-shot analysis):

bash
evalyn calibrate --metric-id <target-metric> --annotations <annotations-dir>

The --annotations flag points to the directory containing annotation files (created by evalyn annotate).

The output shows alignment metrics:

  • Accuracy: overall agreement rate
  • F1 Score: balanced precision/recall
  • Cohen's Kappa: agreement adjusted for chance
Optimizer comparison
OptimizerFlagSpeedBest For
basic--optimizer basicFast (1 API call)First pass, small annotation sets
ape--optimizer apeMediumExploring many prompt variants
opro--optimizer oproMediumIterative refinement
gepa-native--optimizer gepa-nativeSlowBest quality, built-in token tracking
gepa--optimizer gepaSlowEvolutionary (requires pip install gepa)
evoprompt--optimizer evopromptMediumEvolutionary mutation/crossover of prompts
textgrad--optimizer textgradMediumCritique-revise gradient descent on text
miprov2--optimizer miprov2MediumInstruction + few-shot demo co-optimization
promptbreeder--optimizer promptbreederSlowSelf-referential evolutionary prompt search

Optimizer-specific settings are available as CLI flags (e.g., --opro-iterations, --ape-candidates, --gepa-task-lm, --gepa-reflection-lm, --gepa-max-calls).

If basic doesn't improve alignment enough:

bash
evalyn calibrate --metric-id <target-metric> --annotations <annotations-dir> --optimizer gepa

Step 3: Re-evaluate with Calibrated Judges

bash
evalyn run-eval --dataset <dataset-path> --use-calibrated

The --use-calibrated flag loads optimized prompts from the calibrations/ directory.

Step 4: Compare Results

bash
evalyn compare --run1 <original-run-id> --run2 <calibrated-run-id>

Check whether calibrated pass rates better match your expectations.

Step 5: Assess and Iterate

View calibration history:

bash
evalyn list-calibrations

If alignment improved: calibration successful. The judges now match your expectations.

If alignment is still poor:

  • Try a different optimizer (gepa for maximum quality)
  • Add more annotations (more data = better calibration)
  • Review the metric's rubric in the metrics JSON file - it may need manual editing
  • Run evalyn cluster-misalignments --run-id <run-id> to see patterns in disagreements

No Automatic Next Step

This is the terminal skill. The user decides:

  • Iterate: add more annotations and re-calibrate
  • Fix agent: if judges are correct and agent has real issues
  • Expand testing: evalyn simulate --dataset <path> --modes similar,outlier then re-evaluate
  • Ship: export final report with evalyn export --run <id> --format html

Security

  • Security scanPassed, 100 of 100Pattern checks on the skill's files.
  • Model reviewNot runA model review is scheduled for the next rescan.

See the full security report

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

See evaluation results

Source

Popularity

Repository stars
255
Repository forks
26
7-day trend score
28

Stars and forks are for the whole GitHub repository, not this skill alone. Refreshed 20 May 2026, 5 months ago.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/shihongdev/evalyn/evalyn-calibrate/badge)](https://verified-skill.com/skills/shihongdev/evalyn/evalyn-calibrate)

Something wrong with this skill or its page? Tell us.