itseffi/ai-product-evals/eval-smoke-test
Claude Code and agent skill by itseffi/ai-product-evals
- community
Checking that the SKILL.md file is still in the repository.
Guides fast sanity-checking of the eval pipeline. Use when the repo seems broken, CI is failing, provider setup is uncertain, or you need to distinguish runner issues from evaluator or dataset issues.
Install
npx vskill@latest install itseffi/ai-product-evals/eval-smoke-testSKILL.md
The same SKILL.md was published earlier as itseffi/ai-product-evals/eval-smoke-test.
Eval Smoke Test
Overview
- Verify provider availability and selected model configuration before changing eval logic.
- Run the smallest suite first:
evals/quick-test.json. - Separate provider failures, runner failures, evaluator failures, and bad assertions.
- Re-run with
--skip-judgeto isolate judge-related failures from core pipeline failures. - Read traces before editing the suite or the workflow.
Prerequisites
Confirm at least one provider is configured. Inspect run-eval.mjs, evaluators/index.mjs, .github/workflows/eval.yml, and evals/quick-test.json before making changes. If CI is failing, verify whether the failure is in provider setup, model selection, or test logic.
Core Instructions
Start With The Smallest Working Surface
Use the repo’s smoke suite first:
If needed, run against one explicit hosted provider:
Classify The First Failure
Determine which layer fails first:
- Provider failure: missing API key, unavailable local model, invalid model ID
- Runner failure: CLI parsing, provider selection, trace writing, output generation
- Evaluator failure: scoring logic, parsing logic, incorrect judge behavior
- Dataset/assertion failure: the model output is reasonable but the assertion is wrong or too brittle
Use --skip-judge As A Diagnostic Tool
--skip-judge is useful for smoke tests because it removes judge-model dependence while keeping deterministic checks active. If deterministic checks pass but full runs fail, the likely problem is judge configuration or judge prompt design rather than the main eval runner.
Inspect Traces Before Editing
Read traces/ outputs and compare:
- prompt
- model response
- parsed evaluation result
- error type
Do not strengthen or weaken assertions before looking at what the model actually returned.
Repo Files To Inspect
evals/quick-test.jsonrun-eval.mjsevaluators/index.mjsproviders/.github/workflows/eval.ymltraces/
Anti-Patterns
- Starting with the largest benchmark instead of the smoke test.
- Treating missing credentials as model regressions.
- Debugging judge quality before confirming basic execution works.
- Editing assertions before reading traces.
- Adding more test cases before the pipeline is stable.
Security
- Security scanPassed, 90 of 100Pattern checks on the skill's files.
- Model reviewPassed
Quality evaluation
- Task evaluationNot runNo evaluation yet. Run one in Skill Studio.
Source
- SKILL.md in the repositoryCheckingskills/eval-smoke-test/SKILL.md
Popularity
- 7-day trend score
- 5
Not refreshed yet.
Works with
Any agent vskill installs to: 56 targets in all. These 39 have a page on this site.
- Claude Code
- Cursor
- GitHub Copilot
- Windsurf
- Codex
- Gemini CLI
- Cline
- Amp
- Roo Code
- Goose
- Continue
- Kilo Code
- Junie
- Trae
- OpenHands
- Qwen Code
- AI Agent
- Tabnine Chat
- Sourcegraph Cody
- Amazon Q
- Firebase Genkit
- Sweep
- Devin
- Phind
- Mentat
- Claude Desktop
- Aider
- GPT Pilot
- Plandex
- bolt.new
- v0
- Replit Agent
- Anse
- TypingMind
- BetterChatGPT
- LibreChat
- Warp AI
- Pieces
- Fig AI
Embed badge
[](https://verified-skill.com/skills/itseffi/ai-product-evals/eval-smoke-test)