socialadsmentor/sam-agentic-harness/ai-regression-testing
Claude Code and agent skill by socialadsmentor/sam-agentic-harness
- community
Checking that the SKILL.md file is still in the repository.
Regression testing strategies for AI-assisted development. Sandbox-mode API testing without database dependencies, automated bug-check workflows, and patterns to catch AI blind spots where the same model writes and reviews code.
Install
npx vskill@latest install socialadsmentor/sam-agentic-harness/ai-regression-testingSKILL.md
The same SKILL.md was published earlier as socialadsmentor/sam-agentic-harness/ai-regression-testing.
AI Regression Testing
Testing patterns specifically designed for AI-assisted development, where the same model writes code and reviews it - creating systematic blind spots that only automated tests can catch.
When to Activate
- AI agent (Claude Code, Cursor, Codex) has modified API routes or backend logic
- A bug was found and fixed - need to prevent re-introduction
- Project has a sandbox/mock mode that can be leveraged for DB-free testing
- Running
/bug-checkor similar review commands after code changes - Multiple code paths exist (sandbox vs production, feature flags, etc.)
The Core Problem
When an AI writes code and then reviews its own work, it carries the same assumptions into both steps. This creates a predictable failure pattern:
A common shape: a fix lands in one code path (say, an API response), the self-review passes because it shares the same blind spot as the fix, and the same class of miss (a forgotten query column, a duplicated sandbox-vs-production path) resurfaces across several follow-up "fixes" until an actual test run - not another self-review - catches it.
The pattern: sandbox/production path inconsistency is a common AI-introduced regression.
Sandbox-Mode API Testing
Most projects with AI-friendly architecture have a sandbox/mock mode. This is the key to fast, DB-free API testing.
Setup (Vitest + Next.js App Router)
Test Helper for Next.js API Routes
Writing Regression Tests
The key principle: write tests for bugs that were found, not for code that works.
Testing Sandbox/Production Parity
The most common AI regression: fixing production path but forgetting sandbox path (or vice versa).
Integrating Tests into Bug-Check Workflow
Custom Command Definition
The Workflow
Common AI Regression Patterns
Pattern 1: Sandbox/Production Path Mismatch
Frequency: Most common (observed in 3 out of 4 regressions)
Test to catch it:
Pattern 2: SELECT Clause Omission
Frequency: Common with Supabase/Prisma when adding new columns
Pattern 3: Error State Leakage
Frequency: Moderate - when adding error handling to existing components
Pattern 4: Optimistic Update Without Proper Rollback
Strategy: Test Where Bugs Were Found
Don't aim for 100% coverage. Instead:
Why this works with AI development:
- AI tends to make the same category of mistake repeatedly
- Bugs cluster in complex areas (auth, multi-path logic, state management)
- Once tested, that exact regression cannot happen again
- Test count grows organically with bug fixes - no wasted effort
Quick Reference
| AI Regression Pattern | Test Strategy | Priority | |
|---|---|---|---|
| Sandbox/production mismatch | Assert same response shape in sandbox mode | High | |
| SELECT clause omission | Assert all required fields in response | High | |
| Error state leakage | Assert state cleanup on error | Medium | |
| Missing rollback | Assert state restored on API failure | Medium | |
| Type cast masking null | Assert field is not undefined | Medium |
DO / DON'T
DO:
- Write tests immediately after finding a bug (before fixing it if possible)
- Test the API response shape, not the implementation
- Run tests as the first step of every bug-check
- Keep tests fast (< 1 second total with sandbox mode)
- Name tests after the bug they prevent (e.g., "BUG-R1 regression")
DON'T:
- Write tests for code that has never had a bug
- Trust AI self-review as a substitute for automated tests
- Skip sandbox path testing because "it's just mock data"
- Write integration tests when unit tests suffice
- Aim for coverage percentage - aim for regression prevention
Security
- Security scanPassed, 95 of 100Pattern checks on the skill's files.
- Model reviewPassed
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingskills/ai-regression-testing/SKILL.md
Popularity
- 7-day trend score
- 0
Not refreshed yet.
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/socialadsmentor/sam-agentic-harness/ai-regression-testing)