sickn33/agentic-awesome-skills/ai-prompt-regression-testing
Claude Code and agent skill by sickn33/agentic-awesome-skills
- community
Checking that the SKILL.md file is still in the repository.
Use when comparing prompt changes against a fixed evaluation set: record actual model settings, required checks, missing results, judge limitations and case-level regression evidence.
Install
npx vskill@latest install sickn33/agentic-awesome-skills/ai-prompt-regression-testingSKILL.md
The same SKILL.md was published earlier as sickn33/agentic-awesome-skills/ai-prompt-regression-testing.
AI Prompt Regression Testing
When to Use
- Compare a prompt change with an observed baseline on a fixed evaluation set.
- Detect schema, factuality, refusal, tool-selection or task-completion regressions.
- Record a reviewable decision when results are noisy, missing or contradictory.
Workflow
- Define the task, representative cases and forbidden outcomes. Use synthetic
or approved/redacted data; freeze case IDs and the dataset revision before runs.
- Record baseline/candidate prompt digests, exact available model/version,
inference settings, tool versions and evaluation environment. Do not hardcode old model-family options or silently change models between comparisons.
- Predeclare checks and acceptance rules. Use deterministic schema/field checks
for structured output, behavioral checks for tools and a clear human/rubric standard for open text. Similarity does not prove factual or code correctness.
- Run matched baseline/candidate cases under the same permissions and budget.
Record latency/cost and repeated runs when variability matters. A timeout, missing output or judge error is incomplete, not a pass or a zero-cost result.
- Treat judge outputs as evidence with limitations. Blind/randomize presentation
where feasible, calibrate against human-labeled examples, and record judge identity/rubric. Do not ask a model to verify private reasoning or accept a judge's confidence as an objective correctness measurement.
- Review differences case by case and report the denominators, failures and
uncertainty. Keep mandatory safety/schema failures visible even if aggregate similarity improves. Use the predeclared rule, not a threshold selected after results.
- Store a redacted report and exact artifacts at an authorized destination.
Decide retain/revise/reject for the stated dataset; deployment is separate.
Regression register
| Field | Type | Example / meaning | |
|---|---|---|---|
| case_id | string | PTEST-001; a textual ID, not SERIAL | |
| dataset_revision | immutable reference | exact case inventory, expected checks and labels | |
| prompt_revision | string | baseline/candidate digest | |
| model_runtime | string | exact available ID/version plus settings | |
| check | string | required JSON fields or approved rubric criterion | |
| result | enum | pass / fail / incomplete | |
| evidence | reference | redacted actual output, tool trace or observed check | |
| judge_configuration | optional object | exact judge ID/rubric/version and calibration limits | |
| latency_ms | nonnegative number or null | null if not observed; never invented | |
| cost | amount/currency or null | actual billing basis; unknown remains unknown | |
| run_revision | immutable reference | binds the report to the actual run |
IDs and statuses must have consistent SQL/JSON/CSV representations. Do not store private prompts, user messages or tokens in a public spreadsheet or Notion register.
Worked example
Two cases test a support-answer prompt. PTEST-001 requires an answer grounded in the supplied policy; PTEST-002 requires a structured escalation when the policy is absent. The candidate sounds more helpful but PTEST-002 has no observed result. The report is incomplete, not a win. The following Python 3 helper aggregates already reviewed case statuses; it does not call models or decide correctness.
Expected output: incomplete with reason missing or incomplete case. For all observed pass statuses the output is checks-passed, meaning only those recorded checks passed; it is not a release or general quality certificate.
Limitations
- No RPC, relayer, signing key or blockchain deployment is required for this
generic prompt workflow. Real integrations may have their own dependencies.
- Exact text match is unsuitable for many valid open-ended outputs; embeddings
and judges have blind spots and cannot replace mandatory behavioral checks.
- A golden set must be reviewed/versioned; it is not created merely by naming it.
- This entry does not include a provider adapter, judge, dataset or runnable suite.
Use the project's supplied materials and state what was actually executed.
- A finite test set cannot prove future prompt behavior or absence of injection.
Security
- Security scanPassed, 100 of 100Pattern checks on the skill's files.
- Model reviewNot runA model review is scheduled for the next rescan.
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingskills/ai-prompt-regression-testing/SKILL.md
Popularity
- Repository stars
- 47,174
- Repository forks
- 6,873
- 7-day trend score
- 54
Stars and forks are for the whole GitHub repository, not this skill alone. Not refreshed yet.
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/sickn33/agentic-awesome-skills/ai-prompt-regression-testing)