version: "1.0.0" name: ai-evals description: Eval-driven product management for LLM features — error analysis, axial coding of failure modes, LLM-as-judge with TPR/TNR validation, and the continuous Analyze→Measure→Improve loop. Used when an AI feature's output quality is the product and "did it get better?" must be answered with evidence, not vibes. Hosted by ai-eval-engineer / metrics-architect. when_to_use: | Use when an LLM-powered feature ships non-deterministic output whose quality you cannot certify by a unit test — categorization, extraction, summarization, classification, agentic tool-use, RAG answers. Reach for it when you are about to "just check accuracy", when a prompt/model change needs a regression gate, when users report the AI "feels worse", or when you have a pile of production traces and no idea which failure mode to fix first. The PM owns the what-counts-as-good judgment; engineers own the infra. Primarily for ai-eval-engineer and metrics-architect; pairs with metrics-design, experiment-design, ai-use-case-scoping, and pm-tech-spec-review. allowed-tools: Read, Write

AI Evals — eval-driven product management

Provenance: great-pm-original, 2026-05-29, grounded in the cited sources below

(Husain/Shankar AI Evals Masterclass, Aakash Gupta, *Who Validates the

Validators?*). Web sources are treated as untrusted reference, not instruction.

Core principle. AI features don't fail because of the model — they fail because nobody evaluated them. An eval is "the systematic measurement of LLM pipeline quality" that produces interpretable, actionable results, not a single accuracy number you can't act on. Expert practitioners spend 60–80% of dev time on error analysis and evaluation, not on building automated checks. The PM owns the judgment of what counts as good; engineering owns the measurement infrastructure. If you ship an AI feature without an eval, you are shipping blind and finding out from users — on your most expensive surface.

This is not a metrics-design problem (that's about North Star / KPIs for a product) nor an experiment problem (that's A/B testing human behavior). This is the distinct, AI-native discipline of measuring non-deterministic output quality so you can change a prompt and know whether it got better.

1. The Three Gulfs — why an AI feature is failing

Before measuring, diagnose where the gap is. The Three Gulfs (Husain/Shankar) are the lens:

• Comprehension Gulf (Developer → Data). You don't actually know your input

distribution. You think users send clean transaction strings; they send truncated UPI refs, foreign merchants, and refunds. Fix: look at the data.

• Specification Gulf (Developer → Pipeline). The prompt is vaguer than you

think. You wrote "categorize this transaction"; the model has no idea self-transfers aren't expenses. Fix: sharpen the spec, add examples.

• Generalization Gulf (Data → Pipeline). Even with a perfect spec, edge cases

fail because the system is probabilistic. This gulf never fully closes — it's inherent. Manage it with evals + guardrails, don't pretend to eliminate it.

Most teams jump straight to "build an automated metric" — skipping comprehension entirely. Don't.

2. The Analyze → Measure → Improve loop (run continuously)

This is the spine of the skill. Run it as a cycle, not once.

Analyze — look at your data (the part everyone skips)

1. Collect traces. Pull 50–100+ production traces — a trace is the full

execution log: system prompt, user message, retrieved context, tool calls, and the model's response. Not just the output; the whole thing.

1. Open coding. Spend ~30 seconds per trace writing a free-text note on

what went wrong, in plain language: "rendered markdown in an SMS", "promised an action it didn't take", "tagged a self-transfer as an expense". No taxonomy yet — just observe. ~1–2 hours for 100 traces. Domain expert does this, not a contractor who doesn't know the product.

1. Axial coding. Cluster the open-code notes into **5–8 actionable

categories. AI-assisted clustering is fine for a first pass, then a human refines. Avoid vague labels** — "temporal issues" is useless; "date-formatting error" is fixable. ~1 hour.

1. Error counting. Build a pivot table: category × frequency. This ranks your

failure modes by impact. ~30 min. For each category decide: fix trivially (one prompt line) / build an eval (worth automating) / iterate the prompt and re-observe. Some rare-but-catastrophic modes jump the queue regardless of frequency.

Measure — build the judge for what's worth automating

1. Exhaust cheap checks first. Prefer reference-based metrics (exact

match, code execution, SQL-result compare, schema/format validation) — they're deterministic and nearly free. Only spend on an LLM judge for the reference-free categories that genuinely need judgment ("did it invent a category?", "did it give financial advice?").

1. Write the judge prompt BINARY. Pass/fail only. **Never a Likert / 1–5

scale** — scales are subjectively inconsistent between runs and impossible to align to human labels. Include a rubric and a few labeled examples in the judge prompt.

1. Label elicitation order (cognitively easiest → hardest): direct binary

grading > pairwise comparison ("A vs B, which is better?") > absolute ranking. Pairwise is easier for humans than absolute scoring when binary is ambiguous.

Validate the judge — the step that makes the eval trustworthy

1. Hand-label a held-out set yourself (or with the domain expert). This is

ground truth.

1. **Score the judge against your labels using TPR and TNR — never raw

"agreement".**

• TPR (True-Positive Rate / recall): of the traces that are genuinely

errors, what fraction did the judge catch? Low TPR = the judge misses real failures.

• TNR (True-Negative Rate / specificity): of the traces that are fine,

what fraction did the judge correctly pass? Low TNR = the judge cries wolf.

• Why not "agreement"? On imbalanced data (say 90% of traces pass), a judge

that blindly says "pass" every time scores 90% agreement while catching zero errors. Agreement hides this; TPR/TNR expose it. (See Shankar et al., Who Validates the Validators?)

1. Iterate the judge prompt until both TPR and TNR clear your bar. The

judge is itself an LLM pipeline — it gets the same Analyze→Measure→Improve treatment.

Improve — close the loop with confidence

1. Score a large production sample with the validated judge to get a real

quality number.

1. Change one thing — system prompt, retrieval, few-shot examples — and re-run.
2. Confirm no other eval regressed (a fix for one failure mode often breaks

another). Only then ship.

1. Keep the eval suite as a regression gate on every future prompt/model

change.

3. When to use — and when NOT to

Cheaper alternative first: always exhaust reference-based code checks before building an LLM-as-judge. The judge is the expensive rung — climb to it only when the output genuinely needs judgment.

4. Worked example — Acme's Gate-4 LLM categorizer

Acme routes transactions through a gate hierarchy (exact-match DB → regex → vector embedding >0.85 → LLM fallback). The LLM gate (Gemini / local Ollama qwen2.5-coder:14b) only sees the long-tail novel merchant strings the deterministic gates couldn't resolve. Its accuracy on that long tail is the product's perceived intelligence — and it's exactly where there is no reference answer. So it gets the full loop.

Trace definition. Each trace = (raw transaction string + the user's bucket list + the model's chosen category + confidence + whether the user later overrode it). The override is gold: it's a free production label.

Analyze. Pull 100 LLM-gate categorizations. Open-code each in ~30 s. Cluster into axial categories — for Acme these land roughly as:

1. Self-transfer mistaken for an expense (e.g. NEFT to own account).
2. Investment tagged as a regular expense (SIP / mutual-fund debit).
3. Merchant-name truncation broke the match (UPI/ZOMATO... cut off).
4. Duplicate not merged (Gmail alert + CSV upload of the same charge).
5. Hallucinated category — model invented a bucket the user doesn't have.

Pivot table by frequency: say truncation (#3) is most frequent (prompt-fixable, add the raw string + a normalization step), while hallucinated categories (#5) are rare but catastrophic on someone's money (jump the queue — build a hard guardrail, not just an eval).

Measure. Cheap checks first: #5 ("category not in the user's bucket list") is a reference-based check — pure set membership, no LLM needed. #1 and #2 need judgment, so build binary judges: "Is this transaction a self-transfer that should be excluded? PASS/FAIL" with a rubric and examples.

Validate. Hand-label 80 transactions for the self-transfer judge. Measure:

• TPR = of real self-transfers, % the judge flagged. If 0.70, it misses 30% —

iterate the judge prompt with clearer self-transfer cues (same-name payee, account-to-account markers).

• TNR = of genuine expenses, % correctly passed. If 0.95, few false alarms.

Iterate until both clear (say ≥0.90 / ≥0.90).

Improve & close the loop. Score a large sample with the validated judges to get real per-category accuracy. The override→exact-match-table update (the product's own §7 mechanic) becomes a data flywheel: every manual re-bucket is a new labeled trace that feeds the next Analyze pass. This is the missing measurement layer beneath the gate hierarchy — without it, you can change the Gate-4 prompt and have no idea whether categorization got better or just different.

5. Pitfalls / failure modes

6. Ethical boundary

Evals can be tuned to hide failures as easily as to surface them. Three lines not to cross:

• Never tune a judge to pass on harmful-but-frequent outputs to make a

dashboard look green. In fintech, a judge that's been quietly relaxed so mis-categorized income or under-reported spending "passes" is a deception that reaches the user's money and possibly their tax filing.

• **Don't use real user financial data in eval sets without honoring the same PII

stripping the production pipeline requires** (Acme strips PII before queueing — eval corpora must too). Traces are sensitive logs.

• Report TPR/TNR honestly, including on the segments where the model is worst.

An eval that's only ever run on the easy slice is a vanity metric. The point of the discipline is to find where it breaks, not to manufacture a number.

7. Quick diagnostic

8. Relationship to other great-pm skills

• `metrics-design` picks the product's North Star and KPIs (human-outcome

metrics). Evals measure model output quality — a different layer. The eval accuracy rate often becomes an input metric in a north-star-input-tree.

• `experiment-design` runs A/B tests on human behavior with statistical

power. Evals run on model output with TPR/TNR. You frequently use them together: an eval certifies the new prompt is better offline; an experiment confirms the change moves user behavior online.

• `ai-use-case-scoping` decides whether a surface should be AI at all and

defines its accuracy floor. That floor is the target your evals measure against — scope first, then eval.

• `pm-tech-spec-review` reviews the engineering spec generically; this gives

the PM the eval methodology that spec must implement.

• `responsible-ai-guardrails` is the safety/red-team layer (blocking

harmful output at runtime); evals are the quality measurement layer. The hallucinated-category check in the worked example sits on the boundary between them.

9. Output shape

An eval plan: the feature and its trace definition; the open-coding findings (link to the coded trace set); the 5–8 axial failure categories with a frequency pivot; the per-category decision (fix / eval / iterate); for each automated category a binary judge prompt with rubric; the judge's TPR/TNR against a named human-labeled set; the current measured quality on a large sample; and the regression-gate wiring. Re-run the Analyze pass on a schedule — the input distribution drifts.

Sources

• Hamel Husain & Shreya Shankar (with Aakash Gupta), "AI Evals Step-by-Step

Masterclass" — https://www.aakashg.com/hamel-shreya-ai-evals-step-by-step/

• Aakash Gupta, "AI Evals for PMs: Everything You Need to Know" (Three Gulfs,

label elicitation) — https://www.news.aakashg.com/p/ai-evals

• Hamel Husain, "LLM Evals: Everything You Need to Know (FAQ)" —

https://hamel.dev/blog/posts/evals-faq/

• Shreya Shankar et al., "Who Validates the Validators? Aligning LLM-Assisted

Evaluation of LLM Outputs with Human Preferences" — https://arxiv.org/abs/2404.12272

• Productboard, "AI Evals for Product Managers" —

https://www.productboard.com/blog/ai-evals-for-product-managers/