mhylle/claude-skills-collection/verification-loop

Verification Loop

<!-- contents: generated by tests/skill_structure.py --write; edit headings, not this block -->

Contents

• Design philosophy
• When to use
• Project type detection
• Check 1 — Build verification
• Check 2 — Type verification
• Check 3 — Lint verification
• Check 4 — Test verification
• Check 5 — Security scan
• Check 6 — Diff review
• Integration with implement-phase
• Standalone invocation
• Configuration
• Best practices
• Troubleshooting
• Appendix: quick reference
• Related skills

Reference files (open one when a step points to it):

• references/go.md: Verification commands — Go
• references/nodejs-typescript.md: Verification commands — Node.js / TypeScript
• references/python.md: Verification commands — Python
• references/rust.md: Verification commands — Rust

<!-- /contents -->

A systematic 6-check verification framework that validates implementation quality across multiple dimensions: build, types, lint, tests, security, and diff review. Catches issues early, ensures code compiles, type-checks, passes linting, runs tests, has no security issues, and contains no unintended changes.

Terminology: this skill uses "Checks" (Check 1–6). Don't confuse with "Phases" (plan phases) or "Steps" (implement-phase steps).

Design philosophy

Defense in depth — each check catches a different category of issue:

Fail fast — checks are ordered by detection speed. Build errors appear in seconds; security scans take longer. Fast checks first provide rapid feedback on common issues.

Project-agnostic — detect project type automatically, apply the matching tooling. Node.js, Python, Go, Rust, and mixed-language projects all work.

Idempotent — running the loop multiple times produces the same result. Each check passes or fails deterministically based on codebase state.

When to use

Automatically invoked by implement-phase as Step 2 (Exit Condition Verification), before integration testing, after implementation subagents complete.

Manually invoked for:

• Validating changes before committing
• Checking code quality after refactoring
• Pre-merge verification
• Reproducing CI/CD failures locally

Don't use for:

• Reading or analyzing code (no changes made)
• Running exploratory tests (use the test runner directly)
• Checking a single file (use individual tools)

Project type detection

Detect project type first, load the matching reference file, then run the 6 checks with those commands.

Detection logic:

detect_project_type() {
  if [ -f "package.json" ]; then
    if [ -f "tsconfig.json" ]; then echo "typescript"; else echo "nodejs"; fi
  elif [ -f "pyproject.toml" ] || [ -f "setup.py" ]; then echo "python"
  elif [ -f "go.mod" ]; then echo "go"
  elif [ -f "Cargo.toml" ]; then echo "rust"
  else echo "unknown"; fi
}

After detection, only load the matching reference file — don't load all four, that's wasted context.

Check 1 — Build verification

Purpose: ensure the codebase compiles, bundles, and produces valid artifacts.

PASS: build command exits 0, no compilation errors, all artifacts generated, no missing-dependency errors.

FAIL: non-zero exit code, compilation/bundling errors, missing or incomplete artifacts, unresolved deps.

Commands → look up the matching language reference file (references/<lang>.md).

Failure handling: parse the error output → identify root cause (missing import? syntax error? type mismatch?) → spawn fix subagent with error context → re-run build. Max 3 retries.

Output shape:

CHECK_1_BUILD_VERIFICATION:
  STATUS: PASS | FAIL
  COMMAND: [command run]
  EXIT_CODE: 0 | [non-zero]
  DURATION: 12.3s
  ARTIFACTS: [list]
  ERRORS: [] | [list of errors]

Check 2 — Type verification

Purpose: ensure type safety across the codebase with no type errors.

PASS: type checker exits 0, no type errors, all assertions valid, no missing type definitions.

FAIL: type errors, missing type definitions for dependencies, invalid type assertions, unreachable code (in strict mode).

Commands → language reference.

Failure handling: parse errors → categorize (missing types → add annotations; type mismatch → fix types; missing definitions → install @types/* or stubs) → spawn fix subagent → re-run. Max 3 retries.

Output shape:

CHECK_2_TYPE_VERIFICATION:
  STATUS: PASS | FAIL
  COMMAND: [command run]
  EXIT_CODE: 0 | [non-zero]
  DURATION: 8.2s
  FILES_CHECKED: [count]
  ERRORS: [] | [ { file, line, error } ]

Check 3 — Lint verification

Purpose: ensure code follows style guidelines and catches potential bugs.

PASS: linter exits 0, no errors (warnings may be acceptable by config), all auto-fixable issues resolved, formatting matches project standards.

FAIL: lint errors, unfixable formatting issues, security-related lint rules violated, complexity thresholds exceeded.

Commands → language reference. Always run auto-fix first, then re-run the check without it and assert clean — this resolves mechanical issues before flagging real ones.

Failure handling: auto-fix → parse remaining errors → spawn fix subagent for non-auto-fixable issues → re-run. Max 3 retries.

Output shape:

CHECK_3_LINT_VERIFICATION:
  STATUS: PASS | FAIL
  COMMANDS: [
    { cmd: [lint-fix], exit_code: 0 },
    { cmd: [lint-check], exit_code: 0 }
  ]
  AUTO_FIXED: [count]
  REMAINING_ERRORS: 0 | [ { rule, file, line } ]
  WARNINGS: [count]

Check 4 — Test verification

Purpose: ensure all tests pass and new code has appropriate coverage.

PASS: every test project reported, all tests pass (exit 0) in every stability run, no unjustified skips, coverage thresholds met (if configured).

FAIL: any test failure, a test project that produced no result (NOT RUN), a test whose result differs between runs (FLAKY), coverage below threshold, test timeout, test infrastructure errors.

Commands → language reference.

Account for every test project. List the test projects first (for .NET from the .sln or dotnet sln list; for workspaces and monorepos from the workspace config) and compare them with the projects that printed a result. Report each missing project as NOT RUN, then run it on its own and report its counts. A run is green only when every test project reported: dotnet test <sln> skips a test project when a project it references fails to run and prints nothing for it, so summing the summary lines reports a green-looking subset.

Check stability. Run the full suite three times by default, and five times when the change touches performance, allocation, timing or concurrency tests. Report every test whose result differs between runs as FLAKY, with its fail/pass counts. When one full run takes longer than about ten minutes, run the full suite once and repeat only the test projects the change touched, plus any that failed or were flaky.

Failure handling: identify failing tests → categorize (test bug → fix the test; implementation bug → fix the code; environmental → fix test setup) → spawn fix subagent → re-run failing tests first (faster feedback), then full suite after fix. Max 3 retries.

Output shape:

CHECK_4_TEST_VERIFICATION:
  STATUS: PASS | FAIL
  COMMAND: [command run]
  EXIT_CODE: 0 | [non-zero]
  DURATION: 45.2s
  TESTS_RUN: [count]
  TESTS_PASSED: [count]
  TESTS_FAILED: [count]
  TESTS_SKIPPED: [count]
  TEST_PROJECTS: [reported]/[expected]
  NOT_RUN: [] | [ { project, counts when run alone } ]
  STABILITY_RUNS: [count]
  FLAKY: [] | [ { test, failed, passed } ]
  COVERAGE: [percent]
  FAILURES: [] | [ { test, file, error } ]

Check 5 — Security scan

Purpose: detect secrets, debug code, and security vulnerabilities before code reaches production.

PASS: no hardcoded secrets, no console.log/print in production code, no debug flags enabled, no known vulnerable deps (if scanned).

FAIL: secrets detected (API keys, passwords, tokens), debug code in production paths, debug flags left enabled, critical vulnerabilities in deps.

5a. Secrets detection

Use whichever tool is installed; pick the first available:

# git-secrets
git secrets --scan
 
# gitleaks
gitleaks detect --source .
 
# trufflehog
trufflehog filesystem .
 
# Manual patterns (fallback)
grep -r "API_KEY\|SECRET\|PASSWORD\|TOKEN" src/ --include="*.ts" --include="*.js"
grep -r "sk-\|pk_\|api_key\|secret_key" src/

Common secret patterns:

# AWS
AKIA[0-9A-Z]{16}
aws_secret_access_key
 
# API Keys (generic)
[aA][pP][iI]_?[kK][eE][yY].*=.*['"][a-zA-Z0-9]{20,}['"]
 
# Private Keys
-----BEGIN (RSA|DSA|EC|OPENSSH) PRIVATE KEY-----
 
# Tokens
(ghp|gho|ghu|ghs|ghr)_[A-Za-z0-9_]{36,}     # GitHub
xox[baprs]-[0-9a-zA-Z-]+                    # Slack

5b. Console / debug code detection

Language-specific — see the matching reference file. Allowed exceptions: logger framework calls (logger.info, logger.debug), error handling in catch blocks (if configured), test files, development-only files.

5c. Debug flag detection

# Environment checks
grep -rn "NODE_ENV.*development\|DEBUG.*true\|SKIP_AUTH" src/
 
# Disabled security
grep -rn "verify.*false\|secure.*false\|https.*false" src/
 
# TODO/FIXME with security implications
grep -rn "TODO.*security\|FIXME.*auth\|HACK.*bypass" src/

5d. Dependency vulnerability scan

Per-language commands in the matching reference (npm audit, pip-audit, govulncheck, cargo audit).

Failure handling

• Secrets detected → CRITICAL. Must remove before proceeding.
• Console logs → WARNING. Auto-remove or justify with logger.
• Debug flags → WARNING. Must disable for production.
• Vulnerabilities → varies by severity (CRITICAL blocks; others warn).

Output shape:

CHECK_5_SECURITY_SCAN:
  STATUS: PASS | FAIL
  SCANS_RUN: [
    { scan: "secrets", status: "PASS", findings: 0 },
    { scan: "console_logs", status: "WARN", findings: 3 },
    { scan: "debug_flags", status: "PASS", findings: 0 },
    { scan: "dependencies", status: "PASS", findings: 0 }
  ]
  CRITICAL_FINDINGS: 0 | [count]
  WARNINGS: [count]
  DETAILS: [] | [ { type, file, line } ]

Check 6 — Diff review

Purpose: verify only intended changes are present; no unintended modifications.

PASS: all changes relate to the current phase/task, no unrelated file modifications, file count within expected range, no accidental deletions or additions, no formatting-only changes to unrelated files.