calibre-labs/reforge-ai-evals/uig

User Input Grid Skill

A User Input Grid (UIG) — also called a Synthetic Query Matrix (SQM) — is a structured framework for ensuring eval coverage of an AI product reflects the real diversity of users, content domains, input types, and input styles the system will face in production. It comes from the Reforge AI Evals curriculum and a few teams at Amplitude, Linear, and Notion have published versions of it.

The core idea: pick 3–5 dimensions that drive meaningful diversity in your product, define 4–7 values per dimension, and treat the cartesian product (pruned for plausibility) as your eval sourcing matrix. A 4-dimension grid with average ~5 values per dimension gives ~625 theoretical combinations — practical eval sets land at 30–80 high-coverage inputs after pruning.

The reason this matters: most AI eval datasets are built from one of three weak sources — engineer intuition (overfits to features the engineer knows about), real-user logs (overfits to common easy cases users actually try), or synthetic generation without a framework (random uneven coverage). A UIG forces you to be deliberate about what you're testing, makes gaps visible, and gives you a shared vocabulary with the product team for talking about coverage.

When to use this skill

This skill triggers in three modes — be explicit about which one you're in before starting:

1. Greenfield UIG — designing a UIG for a new AI product/feature with no existing dataset.

Workflow ends at "framework + example inputs to seed the dataset."

1. Audit existing dataset(s) — there are one or more eval datasets and you want to know what

they're missing. Workflow includes evaluating each dataset against the UIG with STRONG/PARTIAL/WEAK coverage ratings, surfacing gaps, and recommending additions.

1. Both — design the UIG informed by patterns in the existing dataset, then audit the dataset

against the UIG. This is the most common case when a team has been building evals ad-hoc and wants to put structure around them.

If the mode isn't specified, ask in one short question, then proceed.

Workflow

Step 1 — Understand the product

Before designing dimensions, you need to know: what does this AI product do, who uses it, what data does it have access to, what tools does it call, and what does success look like? Don't skip this — a UIG built without product understanding produces generic dimensions that don't catch real failure modes.

If the user has shared product docs, sample inputs, system prompts, or tool definitions, read them first. If you don't have enough context, ask 2–3 specific questions — not a generic interview. Examples: "What actions can the AI take? Are there topics it explicitly can't handle? What are the most common user goals?"

Step 2 — Read the dataset(s) if any exist

If there are existing datasets, read them before designing the UIG. Real input data is the best signal you have. Look for:

• Persona signals — are there user-type labels in metadata? Do inputs imply different user

contexts (new user, power user, admin, customer vs. employee)?

• Domain coverage — what topic areas do inputs touch? What's missing from the inputs that

the product clearly supports?

• Input type distribution — how many action instructions vs. information requests vs.

multi-step tasks vs. ambiguous inputs?

• Style patterns — are inputs well-specified, vague, count-only, edge cases?
• Scoring criteria quality — are pass/fail criteria deterministic, or vague LLM-judge prompts?

Watch for OR conditions ("X or asks for clarification") that make the test always pass.

This "dataset-first reading" surfaces dimensions you'd never invent from first principles.

Step 3 — Design the dimensions

Pick 3–5 dimensions. Fewer than 3 and the grid doesn't drive enough diversity; more than 5 and the cartesian product explodes and the dimensions start overlapping conceptually.

Two dimensions are nearly universal for any AI product. Use these as defaults, adapted to context:

Input Type — what the user is asking the AI to do

For products with workflow-triggered inputs (no user text — an automated record or event fires the AI): add Automated trigger as a value and ensure your eval dataset includes both human-initiated and system-initiated inputs.

For AI products with traditional search/query interfaces, "Action instruction" may be rare and "Lookup" a better fit — adapt the values to what users actually send.

Input Style — how well-formed or specified the input is

Two dimensions are always domain-specific and need to be derived from the product:

• User Persona — different user types whose context changes what a good response looks like.

Don't fall back to "novice / intermediate / expert" — that's a smell. Real personas are defined by their goals and context (e.g. "sales rep closing deals" vs. "service agent handling escalations" vs. "customer self-serving on a portal").

• Data/Content Domain — the distinct topic areas, capability surfaces, or data categories the

product covers. This is the dimension you most need the dataset to inform — first-principles drafts almost always miss something.

For agent products, consider a fifth dimension: Action/Output Type — what the AI produces (drafted email, updated record, routing decision, summary, action refusal). This catches output format and action correctness bugs that input-diversity dimensions alone won't surface.

For each dimension, define 4–7 values. Each value needs a short justification (one sentence on why it's distinct from the others) and a characteristic example input showing what it generates.

Step 4 — Write the UIG markdown file

Ask the user where to save the output. Default: <product-name>-uig.md in the current working directory.

Structure the file as follows:

# [Product Name] User Input Grid
Brief intro paragraph — what product, what the UIG covers, target dataset size.
 
## Dimensions Overview
Summary table: Dimension | Values | Notes
 
## [Dimension 1] — [Name]
Table: Value | Definition | Example Input | Why distinct
Notes on judgment calls made.
 
## [Dimension 2] — [Name]
...
 
## Example Input Combinations
Table of 8–12 rows sampling the grid: one row per combination, with a concrete example input.
This forces you to verify each combination produces a sensible, testable input.
 
## Dataset Evaluation  ← only in modes 2 and 3
Per-dataset coverage tables and overall summary.
 
## Recommendations
Prioritized gap list with example inputs.

The example inputs section is important — it makes the abstract grid concrete and forces you to verify that each combination produces a sensible input. If a combination feels nonsensical when you try to write an input for it, that's a signal the grid has an implausible region you can prune.

Step 5 — Evaluate datasets against the UIG (if mode 2 or 3)

For each dataset, rate coverage on each dimension as STRONG / PARTIAL / WEAK:

• STRONG — most values represented, with clear examples
• PARTIAL — some values present, others noticeably missing
• WEAK — only one or two values covered

For each (dataset × dimension) cell, write a one-paragraph note covering: what's present, what's missing, and the practical implication of the gap. The implication matters more than the gap itself — "no multi-step inputs" is a fact; "the AI could score 95% on this dataset and still be broken when users chain two operations in a single message" is the point.

Seven common failure patterns to watch for:

1. Persona collapse — all inputs written from one user type's perspective; the dataset silently

assumes a single user context even though the product serves multiple roles.

1. Instruction skew — too many freeform instructions, missing information requests, aggregations,

or structured/triggered inputs that the product clearly handles.

1. Advice-without-criteria — eval rows that test for "good response" but provide no falsifiable

scoring criterion. If two evaluators would disagree on pass/fail, the criterion isn't ready.

1. OR-condition trap — scoring criteria phrased as "output contains X or asks for clarification."

This always passes because a confused AI can always ask for clarification. Split into two separate eval rows.

1. Real-input feedback loop — dataset built entirely from real user logs, which overfit to the

common easy cases actual users send. Rare but high-stakes inputs never appear.

1. Missing comparison/trend — time-period and multi-entity inputs are consistently absent because

they're harder to write; they're also where AI products frequently fail.

1. Output-quality vs. input-diversity confusion — hallucination tests (output quality checks)

mixed into a coverage dataset (input diversity checks). These need separate eval strategies.

After per-dataset tables, write an overall coverage summary combining all datasets, then a recommendations section prioritized by impact:

1. Input types that are entirely absent — highest-priority gaps; nothing in the eval suite catches

these failure modes.

1. Cross-dimension combinations that span two gap areas — catch coordination bugs.
2. Persona-differentiated inputs where the persona context meaningfully shapes the right response.
3. Out-of-scope and edge-case inputs that test graceful decline or scope clarification.
4. Output-quality / hallucination tests for high-stakes domains.

For each recommendation, give 3–6 example inputs with dimension labels and what the test should verify. Don't just say "add multi-step inputs" — write the actual inputs.

Step 6 — Expand the dataset (if requested)

If asked to "expand the dataset" or "fill the gaps", generate a fresh markdown file with 30–80 new inputs organized by the gap they fill, each tagged with persona / domain / input type / style. For every input, include an evaluable scoring criterion.

The OR-condition trap: Never write a scoring criterion as "output does X or asks for clarification." That criterion always passes — any AI can deflect with a clarifying question. Instead, split into two rows: one where input is clear and the AI must act, one where input is genuinely ambiguous and graceful clarification is the right response.