kienbui1995/magic-powers/ai-harness
Claude Code and agent skill by kienbui1995/magic-powers
Trust tier 3
- community
Source check pending
Checking that the SKILL.md file is still in the repository.
Use when building evaluation infrastructure for AI systems — test harnesses, CI pipelines for AI, automated regression detection, golden datasets, and continuous quality measurement.
Install
npx vskill@latest install kienbui1995/magic-powers/ai-harnessSKILL.md
The same SKILL.md was published earlier as kienbui1995/magic-powers/ai-harness.
AI Harness
When to Use
- Setting up automated evaluation pipeline for an LLM feature
- Catching quality regressions before they reach production
- Building a continuous eval CI/CD system
- Creating golden datasets for systematic testing
- Comparing prompt versions or model upgrades systematically
Core Components of an AI Harness
1. Golden Dataset Design
- Representative inputs covering: happy path, edge cases, adversarial, real user examples
- Each test case: input + expected output + evaluation criteria
- Minimum 50-100 cases before automated eval is meaningful
- Label types: exact match, semantic similarity, rubric-based, human preference
- Dataset versioning (git or DVC) — same discipline as code
2. Evaluation Layers
Three layers for comprehensive coverage:
3. CI Integration
4. Regression Detection
- Baseline: capture current pass rates per category
- Alert when: pass rate drops >5% vs baseline, any critical test fails, cost increases >20%
- Track over time: eval results → time-series DB → dashboard
5. Prompt Version Testing
6. Tools and Frameworks
| Tool | Use case | |
|---|---|---|
| LangSmith | Tracing + eval for LangChain apps | |
| Braintrust | Eval platform, dataset management, CI integration | |
| PromptFoo | Open-source prompt testing, CLI-first | |
| Weights & Biases | Experiment tracking, model comparison | |
| Ragas | RAG-specific evaluation (faithfulness, answer relevance) | |
| pytest + custom | Lightweight eval for simple use cases |
Core Jobs
- Golden dataset — curate representative test cases (happy path, edge cases, adversarial, real user queries)
- Eval layers — deterministic checks → model-as-judge → human review
- CI integration — run eval on every PR; block merge on critical failures
- Regression detection — baseline pass rates; alert on >5% drop by category
- Prompt A/B testing — compare prompt versions against dataset; select winner by metric
Key Outputs
- Eval report: pass rate by category + comparison vs baseline
- CI check: green/red with failure breakdown
- Dataset: versioned JSONL with input/expected/criteria
- Regression alert: which category dropped and by how much
Anti-Patterns
- Evaluating only on training distribution (test on real user queries)
- Using BLEU/ROUGE for open-ended generation (meaningless for chat)
- No baseline — "better" means nothing without comparison
- Running eval only before major releases (too late to catch drift)
- One aggregated score hiding category regressions
Statistical Rigor
Avoid declaring "improvement" without statistical evidence:
Key rules:
- Never compare single-run scores — always run 5-10 times minimum
- Report confidence intervals, not just averages
- Distinguish statistical significance (p<0.05) from practical significance (effect size matters)
- A 1% quality improvement usually isn't worth shipping complexity
Dataset Maintenance
Golden datasets degrade — keep them fresh:
Dataset health signals:
- >20% of tests always pass → too easy, add harder cases
- >10% of tests never pass → broken tests or capability gap, investigate
- Last reviewed >30 days → likely stale, review against recent prod distribution
Integration
- Use with
llm-evaluation(frameworks) andllm-observability(production monitoring) - Use
ai-safety-guardrailsto add safety checks as eval layer - CI eval catches regressions; observability catches production drift
Security
- Security scanPassed, 95 of 100Pattern checks on the skill's files.
- Model reviewPassed
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingskills/ai-harness/SKILL.md
Popularity
- 7-day trend score
- 0
Not refreshed yet.
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/kienbui1995/magic-powers/ai-harness)