po4yka/llm-wiki-skills/llm-wiki-eval-tooling

LLM-Wiki Eval Tooling

Goal

Turn LLM-Wiki usefulness, retrieval quality, grounding, security and maintenance questions into measurable tests, scorecards and repeatable CI/review gates.

Use references/docs/18-evaluation-methodology.md as the reference methodology for metric definitions, experiment design, dataset construction, CI strategy and rollout.

When to use

• The user asks which eval framework, dataset format, scorecard, or CI gate fits their LLM-Wiki (e.g. "should we use Ragas or promptfoo", "what does a retrieval eval set look like").
• The user wants to prove or disprove a claim such as "the wiki saves time" or "hybrid retrieval improved answers" with measurable tests.
• The user needs to design PR-time, nightly, or release-time CI gates for retrieval, grounding, or security regressions.
• The user needs a red-team/prompt-injection test plan or a human calibration workflow for LLM judges.
• Route to llm-wiki-eval instead when the user wants an actual pilot run executed, not tool/format selection.

Inputs

• Wiki path and evaluation scope.
• User claim to test, such as “the wiki saves time” or “hybrid retrieval improved answers”.
• Recent real questions/tasks or production traces.
• Retrieval layer details.
• Model/provider policy.
• Desired tools: Ragas, promptfoo, DeepEval, TruLens, LangSmith, OpenAI Evals/API Evals, Phoenix/Arize, pytrec_eval, custom scripts.
• Security constraints and risk tier.

Procedure

1. Pick evaluation layer

2. Choose the tool stack

Do not require every tool. Pick the smallest stack that answers the user's evaluation question.

3. Build an eval set

Use 10-20 real questions for a pilot and expand only after the first run. For serious evaluation, build 200-300 examples plus a high-stakes slice.

Record each item:

id: ""
question: ""
query_type: exact|conceptual|synthesis|multi-hop|recent|sensitive
expected_answer_traits: []
required_pages: []
required_sources: []
expected_passages: []
forbidden_sources: []
required_filters: {}
gold_answer_points: []
freshness_sla_days: 0
sensitivity: public|internal|sensitive|regulated|unknown
risk_tier: low|medium|high|critical
answer_type: factual|synthesis|comparison|decision|debugging|planning
must_cite_sources: true

Use synthetic generation only to expand coverage. Do not use synthetic questions alone to justify adoption.

4. Add qrels and support labels

Use TREC-style qrels for retrieval:

relevance:
  wiki/page.md#heading: 3
  raw/source.pdf#p12: 2
  wiki/related.md#section: 1

Use support labels for grounding:

support_level: extracted|inferred|ambiguous|synthesis|unsupported|conflicting

5. Compare baselines

Use at least two modes when possible:

Do not claim a stack is better unless the eval set represents the user's real work.

6. Define gates by risk tier

Example gates:

retrieval:
  recall_at_10_min: 0.85
  mrr_min: 0.70
  ndcg_at_10_min: 0.80
grounding:
  citation_coverage_min: 0.90
  unsupported_claim_rate_max: 0.05
security:
  prompt_injection_success_rate_max: 0.05
  pii_secret_leaks_max: 0
  cross_tenant_leaks_max: 0
operations:
  stale_tier1_pages_max_pct: 5
  median_review_age_days_max: 7
  broken_verified_links_max: 0

For early pilots, report gates without failing CI until the team agrees thresholds are fair.

7. Plan CI/review workflow

PR-time checks:

• deterministic lint;
• broken links/citations;
• retrieval smoke set;
• promptfoo quality subset;
• secret/PII scan;
• no critical unsupported-claim regressions.

Nightly checks:

• full retrieval eval;
• full grounding eval;
• prompt-injection red team;
• stale-page report;
• review backlog report;
• latency/cost trend report.

Release checks:

• with-wiki vs without-wiki comparison;
• retrieval ablations for major index changes;
• human calibration review;
• security sign-off for public/team deployments.

8. Add human calibration

Monthly sample:

• 25 passing examples;
• 25 failing examples;
• 25 judge/human disagreements;
• 25 high-risk examples.

Use a rubric:

correctness: 1-5
completeness: 1-5
groundedness: 1-5
citation_quality: 1-5
actionability: 1-5
risk: low|medium|high|critical
reviewer_notes: ""

Use human review to calibrate LLM judges and thresholds.

9. Report actionably

Every eval report should include:

• what changed;
• what improved;
• what regressed;
• examples of misses;
• likely root causes;
• next fix;
• whether the next fix is schema, retrieval, ingestion, prompt, review process, security or data policy.

10. Use templates

Recommended templates:

• references/templates/retrieval-eval-set.yaml for retrieval qrels and gates.
• references/templates/eval-scorecard.yaml for multi-layer scorecards.
• references/templates/promptfoo-llm-wiki.yaml for CI/rubric/security eval configuration.
• references/templates/llm-wiki-evals.github-actions.yml for a GitHub Actions starter workflow.

Output

## Evaluation plan
 
## Eval set schema
 
## Baselines and ablations
 
## Tooling recommendation
 
## Metrics and gates
 
## CI/review workflow
 
## Human calibration plan
 
## Security/red-team plan
 
## Scorecard
 
## Failure taxonomy
 
## Next skill

Safety gates

• Do not optimize for note count, graph density or other vanity metrics.
• Do not use synthetic questions alone to justify adoption.
• Do not run evals on sensitive material through cloud tools without policy approval.
• Do not hide examples of failed queries; they are the most useful output.
• Do not claim external benchmarks prove the user's local wiki will work.
• Do not rely exclusively on LLM judges; keep human calibration.
• Do not compare eval runs across changed datasets without recording dataset revision.