anastasiyaw/codex-claude-code-config/harness-audit

Harness Audit

Trigger on phrases like: "audit my harness", "evaluate my agent setup", "score my CLAUDE.md", "is my project ready for long-run", "5-subsystem assessment", "what's missing from my project setup", "/harness-audit". Run proactively when joining an unfamiliar codebase that has agent artifacts (CLAUDE.md, .claude/, AGENTS.md) but obvious gaps. Skip for single-file scripts and pure exploration.

Score a project's agent harness across five subsystems and tell the user which evidenced bottleneck to address first. Distinguish an artifact's presence from demonstrated behavior; never present metadata alone as runtime proof.

Source: Five-subsystem framework adapted from Learn Harness Engineering (walkinglabs, MIT). Adapted to our concrete stack: CLAUDE.md, .claude/rules/, PROBLEMS.md, feature_list.json, init.sh, hooks, handoffs, chronicles.

What This Skill Does

Given a project directory, produces a scorecard like this. This example assumes project-xyz delivers features across sessions and uses pull requests:

=== Harness Audit: project-xyz ===
 
Instructions  4/5  ✓ Agent entrypoint and modular rules are documented and used
                   ~ PR review guidance is not found in the inspected evidence
State         2/5  ✓ 3 handoffs preserve some continuation state
                   ✗ No current record locates active scope and deferred work
                   ~ PROBLEMS.md / feature_list.json are suitable conventions, not prerequisites
Verification  3/5  ✓ Documented test command; pytest is configured
                   ~ No current execution receipt supplied
                   ✗ No documented staged validation/proof route
Scope         3/5  ✓ in-scope principle in CLAUDE.md
                   ~ shared-resource policy is unknown; no serialized lane is declared
                   ✗ Definition of Done not explicit
Lifecycle     2/5  ✗ Needed session-boundary entry/stop behavior is not documented or demonstrated
                   ~ Manual cleanup convention exists but not enforced
 
Bottleneck: State (2/5) — no current locator for feature scope and deferred work
 
Priority improvement (only when the user asks for recommendations):
- Record an execution receipt for the existing test command   ↗ Verification evidence

For an audit-only request, this skill produces the scorecard without making changes or running probes merely to convert unknown into a pass. If the user also requested correction or implementation, the scorecard is an intermediate result: return the confirmed findings to the owning task and execute its necessary, authorized reversible fixes, verification and delivery. Do not stop at a report or assign agent-owned fixes back to the user. Preserve the original acceptance criteria and real external/irreversible boundaries.

The Five Subsystems (Our Adaptation)

See references/checklist-per-subsystem.md for per-subsystem concrete checks. See references/scoring-rubric.md for how to interpret 1-5 scores.

How to Run an Audit

Phase 1 — Gather

Read these files in order (skip silently if missing):

1. CLAUDE.md in project root
2. AGENTS.md in project root (some projects use this name)
3. .claude/rules/*.md (project-level rules)
4. .claude/settings.json and .claude/settings.local.json (hooks config)
5. PROBLEMS.md in root
6. feature_list.json in root
7. init.sh in root (and Makefile / package.json scripts as fallback)
8. .claude/handoffs/ (count files, check INDEX.md existence)
9. .claude/chronicles/ (count files)
10. Sample test config: pytest.ini / package.json test script / Cargo.toml

Use Glob + Read for the harness, then inspect the smallest relevant evidence path: a current test/CI receipt, hook execution trace, or sampled state artifact. This is not a broad code review; absence of behavioral evidence is ~ unknown, not ✓ working.

Phase 2 — Score

For each subsystem, first identify the project delivery model and applicable outcomes in references/checklist-per-subsystem.md. Mark every finding as documented, demonstrated, or unknown; score from evidence rather than file presence alone. The canonical files in the table are useful conventions, not universal prerequisites.

• 5 = applicable outcomes are documented, demonstrated, and consistently followed
• 4 = documented and demonstrated with bounded gaps
• 3 = only documented, only demonstrated, materially partial, or unknown at a relevant boundary
• 2 = isolated or stale structure does not meet most applicable outcomes
• 1 = missing or actively harmful

For each subsystem, list:

• ✓ what's present and working
• ✗ what's missing or broken
• ~ partial / unclear

Phase 3 — Identify Bottleneck

The lowest-scoring subsystem is the bottleneck. Even if other subsystems are weaker by absolute count of checks, the lowest score is the one to fix first because it limits the value of the rest.

Tie-breaker (multiple subsystems at same low score): pick the one whose improvement unlocks progress in others. State often wins when the project lacks a durable locator for active scope and unresolved work; do not assume two particular filenames are required.

Phase 4 — Prioritized Improvement Plan

Only if the user requests recommendations, propose the smallest number of independently shippable actions that address the evidenced bottleneck. For each, name the expected evidence and a local template/example if one actually fits. Do not invent effort, score gains, or a fixed number of steps. Do not expand an audit-only request into implementation; when implementation was already requested, continue that owning task after the audit instead of requesting the same authorization again.

Output Format

Use the visual scorecard format shown at the top of this skill. Sections:

1. Header: === Harness Audit: <project-name> === (one line)
2. Scorecard: 5 lines, one per subsystem, with score + ✓/✗ findings
3. Bottleneck: one line naming the subsystem and score
4. Priority improvements: only when requested, with expected evidence + pointer
5. Projected total: optional, only if user asks and the stated evidence supports a bounded projection

Keep the entire output under 50 lines. The user is scanning for next steps, not reading an essay. Detail goes into the per-subsystem checklist file, not the audit output.

What This Skill Is NOT

• Not a code review — does not look at source code quality
• Not a security audit — does not check for vulnerabilities (use /security-review instead)
• Not a broad test runner — does not manufacture a green result from configuration. It may inspect a current CI/test receipt, or run one user-authorized, task-relevant probe when runtime evidence is part of the requested audit
• Not a standalone implementation methodology — an audit-only request ends with its findings; an audit inside an already authorized repair returns control to that repair, not to user homework
• Not for short-lived work without a durable handoff need — state why the audit is disproportionate instead of applying arbitrary feature/session thresholds

Honest Tradeoffs

• The 5-subsystem framework is opinionated. A project can be perfectly functional with 3 of 5 strong and 2 weak (e.g., a research repo with no lifecycle needs).
• Scoring is subjective at the margins. A 3 vs 4 for "covers basics" is a judgment call. Use the checklist to keep it consistent across audits, not to claim numeric precision.
• The skill assumes our stack conventions. For projects using completely different tooling (e.g., AGENTS.md without .claude/), translate concepts before scoring — don't fail the project on naming.

Related

• Principle 27 (feature-tracking) — full framework explanation
• Principle 01 (harness-design) — Generator-Evaluator pattern, source of "subsystems" thinking
• Templates ${CLAUDE_PLUGIN_ROOT}/templates/long-run-project/ — drop-in files for fixing State + Verification gaps
• Rule rules/long-run-harness.md — convention this audit checks against

Gotchas

• A scorecard example must use the same evidence rules as the checklist; a missing filename alone does not prove a missing outcome.
• Presence of this skill or its rubric is not proof that auditors apply it consistently. Do not cite an example-evaluation file unless it actually exists and was inspected.

Troubleshooting

• Two auditors give different scores to the same evidence: compare the declared delivery model and applicable outcomes, then resolve the checklist/example contradiction; do not add files just to raise the score.
• An authorized repair ends at the scorecard: return each confirmed finding to the original task, implement the necessary correction and verify it. An audit-only request remains read-only.