sickn33/agentic-awesome-skills/ai-prompt-regression-testing

Claude Code and agent skill by sickn33/agentic-awesome-skills

Trust tier 2
  • community
LLM review pending
Source check pending

Checking that the SKILL.md file is still in the repository.

Use when comparing prompt changes against a fixed evaluation set: record actual model settings, required checks, missing results, judge limitations and case-level regression evidence.

Install

npx vskill@latest install sickn33/agentic-awesome-skills/ai-prompt-regression-testing

SKILL.md

The same SKILL.md was published earlier as sickn33/agentic-awesome-skills/ai-prompt-regression-testing.

AI Prompt Regression Testing

When to Use

  • Compare a prompt change with an observed baseline on a fixed evaluation set.
  • Detect schema, factuality, refusal, tool-selection or task-completion regressions.
  • Record a reviewable decision when results are noisy, missing or contradictory.

Workflow

  1. Define the task, representative cases and forbidden outcomes. Use synthetic

or approved/redacted data; freeze case IDs and the dataset revision before runs.

  1. Record baseline/candidate prompt digests, exact available model/version,

inference settings, tool versions and evaluation environment. Do not hardcode old model-family options or silently change models between comparisons.

  1. Predeclare checks and acceptance rules. Use deterministic schema/field checks

for structured output, behavioral checks for tools and a clear human/rubric standard for open text. Similarity does not prove factual or code correctness.

  1. Run matched baseline/candidate cases under the same permissions and budget.

Record latency/cost and repeated runs when variability matters. A timeout, missing output or judge error is incomplete, not a pass or a zero-cost result.

  1. Treat judge outputs as evidence with limitations. Blind/randomize presentation

where feasible, calibrate against human-labeled examples, and record judge identity/rubric. Do not ask a model to verify private reasoning or accept a judge's confidence as an objective correctness measurement.

  1. Review differences case by case and report the denominators, failures and

uncertainty. Keep mandatory safety/schema failures visible even if aggregate similarity improves. Use the predeclared rule, not a threshold selected after results.

  1. Store a redacted report and exact artifacts at an authorized destination.

Decide retain/revise/reject for the stated dataset; deployment is separate.

Regression register

FieldTypeExample / meaning
case_idstringPTEST-001; a textual ID, not SERIAL
dataset_revisionimmutable referenceexact case inventory, expected checks and labels
prompt_revisionstringbaseline/candidate digest
model_runtimestringexact available ID/version plus settings
checkstringrequired JSON fields or approved rubric criterion
resultenumpass / fail / incomplete
evidencereferenceredacted actual output, tool trace or observed check
judge_configurationoptional objectexact judge ID/rubric/version and calibration limits
latency_msnonnegative number or nullnull if not observed; never invented
costamount/currency or nullactual billing basis; unknown remains unknown
run_revisionimmutable referencebinds the report to the actual run

IDs and statuses must have consistent SQL/JSON/CSV representations. Do not store private prompts, user messages or tokens in a public spreadsheet or Notion register.

Worked example

Two cases test a support-answer prompt. PTEST-001 requires an answer grounded in the supplied policy; PTEST-002 requires a structured escalation when the policy is absent. The candidate sounds more helpful but PTEST-002 has no observed result. The report is incomplete, not a win. The following Python 3 helper aggregates already reviewed case statuses; it does not call models or decide correctness.

python
def summarize_regression(results, expected_case_ids):
if not isinstance(expected_case_ids, list) or not expected_case_ids:
return {"status": "incomplete", "reason": "missing case inventory"}
if any(not isinstance(case, str) or not case for case in expected_case_ids):
return {"status": "incomplete", "reason": "invalid case IDs"}
if len(set(expected_case_ids)) != len(expected_case_ids):
return {"status": "incomplete", "reason": "duplicate expected case IDs"}
if not isinstance(results, list) or any(not isinstance(row, dict) for row in results):
return {"status": "incomplete", "reason": "invalid results"}
by_id = {}
for row in results:
case = row.get("case_id")
if not isinstance(case, str) or case not in expected_case_ids or case in by_id:
return {"status": "incomplete", "reason": "unexpected or duplicate result"}
if row.get("result") not in ("pass", "fail", "incomplete"):
return {"status": "incomplete", "reason": "unknown result"}
by_id[case] = row["result"]
failed_cases = [case for case, result in by_id.items() if result == "fail"]
if set(by_id) != set(expected_case_ids) or "incomplete" in by_id.values():
return {"status": "incomplete", "reason": "missing or incomplete case",
"failed_cases": failed_cases, "evaluated": len(by_id)}
return {"status": "regression" if failed_cases else "checks-passed",
"failed_cases": failed_cases, "evaluated": len(by_id)}
print(summarize_regression(
[{"case_id": "PTEST-001", "result": "pass"}],
["PTEST-001", "PTEST-002"],
))

Expected output: incomplete with reason missing or incomplete case. For all observed pass statuses the output is checks-passed, meaning only those recorded checks passed; it is not a release or general quality certificate.

Limitations

  • No RPC, relayer, signing key or blockchain deployment is required for this

generic prompt workflow. Real integrations may have their own dependencies.

  • Exact text match is unsuitable for many valid open-ended outputs; embeddings

and judges have blind spots and cannot replace mandatory behavioral checks.

  • A golden set must be reviewed/versioned; it is not created merely by naming it.
  • This entry does not include a provider adapter, judge, dataset or runnable suite.

Use the project's supplied materials and state what was actually executed.

  • A finite test set cannot prove future prompt behavior or absence of injection.

Security

  • Security scanPassed, 100 of 100Pattern checks on the skill's files.
  • Model reviewNot runA model review is scheduled for the next rescan.

Quality evaluation

  • Task evaluationNot runNo evaluation yet. Run one in Skill Studio.

Source

Popularity

Repository stars
47,174
Repository forks
6,873
7-day trend score
54

Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.

Works with

Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.

  • Claude Code
  • Cursor
  • GitHub Copilot
  • Windsurf
  • Codex
  • Gemini CLI
  • Cline
  • Amp
  • Roo Code
  • Goose
  • Continue
  • Kilo Code
  • Junie
  • Trae
  • OpenHands
  • Qwen Code
  • AI Agent
  • Tabnine Chat
  • Sourcegraph Cody
  • Amazon Q
  • Firebase Genkit
  • Sweep
  • Devin
  • Phind
  • Mentat
  • Claude Desktop
  • Aider
  • GPT Pilot
  • Plandex
  • bolt.new
  • v0
  • Replit Agent
  • Anse
  • TypingMind
  • BetterChatGPT
  • LibreChat
  • Warp AI
  • Pieces
  • Fig AI
Embed badge
vskill VERIFIED badge
[![vskill VERIFIED](https://verified-skill.com/api/v1/skills/sickn33/agentic-awesome-skills/ai-prompt-regression-testing/badge)](https://verified-skill.com/skills/sickn33/agentic-awesome-skills/ai-prompt-regression-testing)

Something wrong with this skill or its page? Tell us.