empiricaai/empirica/code-audit

Code Audit: Noetic Investigation Skill

Investigate code quality. Produce structured remediation plans.

This skill is purely noetic — it discovers, triages, and plans. It does NOT make changes. The output (findings, goals, decisions) feeds directly into the Empirica workflow for any praxic agent to pick up and execute.

How to Run

/code-audit                              # Audit entire project
/code-audit --target src/handlers/       # Audit specific directory
/code-audit --target src/auth.py         # Audit specific file
/code-audit --focus duplication          # Focus on one dimension

Phase 1: Scope

Determine what to audit. If the user specified a target, use it. Otherwise, audit the current project root.

# Determine scope
TARGET="${1:-.}"  # Default to current directory
 
# Quick size assessment
find "$TARGET" -name "*.py" | wc -l          # File count
find "$TARGET" -name "*.py" -exec wc -l {} + | tail -1  # Total LOC

Log the audit scope:

empirica finding-log --finding "Audit scope: $TARGET — N files, N LOC" --impact 0.1

Phase 2: Automated Tool Passes

Run external tools and parse results into findings. Each tool covers a different dimension. Skip any tool that isn't installed — the audit still works without it.

2a. Linting & Style (ruff)

ruff check "$TARGET" --statistics --output-format json 2>/dev/null

Parse results:

• Group by rule category (import order, unused imports, complexity, etc.)
• Log aggregates as findings, not individual violations
• Impact scoring: unused imports = 0.2, complexity violations = 0.5, security = 0.8

# Example: aggregate finding
empirica finding-log --finding "ruff: 23 unused imports across 8 files (F401)" --impact 0.2
empirica finding-log --finding "ruff: 5 functions exceed complexity limit (C901)" --impact 0.5

2b. Dead Code (vulture)

vulture "$TARGET" --min-confidence 80

Parse results:

• Filter out false positives (dynamically called functions, CLI entry points)
• Log confirmed dead code as findings
• Log uncertain cases as unknowns

empirica finding-log --finding "vulture: 12 unused functions detected (80%+ confidence)" --impact 0.4
empirica unknown-log --unknown "vulture flagged handle_legacy_sync() — verify if called dynamically"

2c. Complexity Metrics (radon)

radon cc "$TARGET" -s -a --min C    # Only show functions rated C or worse
radon mi "$TARGET" -s --min B       # Maintainability index

Parse results:

• Functions rated C (11-20) → finding with impact 0.4
• Functions rated D (21-30) → finding with impact 0.6
• Functions rated F (31+) → finding with impact 0.8
• Log the worst offenders by name and file

empirica finding-log --finding "radon: process_vectors() in workflow_commands.py has CC=27 (D)" --impact 0.6

2d. Test Freshness (scripts/test_freshness_audit.py)

python3 scripts/test_freshness_audit.py --human
python3 scripts/test_freshness_audit.py --baseline   # counts only, track direction

Not a coverage check. Coverage asks "was this line executed?"; this asks "could this test have failed?" Tests drift like any other artifact, and a drifted test doesn't go red — it goes green over the wrong contract and guards the defect.

Triage, don't bulk-fix. A minimal fixture is a legitimate choice; what makes it a finding is that the missing columns arrived in a migration after it was written. Log real ones as findings, and record deliberate omissions so the next sweep stays quiet on them.

2e. Type Errors (pyright)

pyright "$TARGET" --outputjson 2>/dev/null

Parse results:

• Count errors by category (missing types, incompatible types, possibly unbound)
• Log aggregates, not individual errors

empirica finding-log --finding "pyright: 45 type errors — 20 missing annotations, 15 incompatible, 10 unbound" --impact 0.3

Phase 3: Structural Review (Manual)

These dimensions require AI judgment — tools can't catch them.

3a. File Size Analysis

find "$TARGET" -name "*.py" -exec wc -l {} + | sort -rn | head -20

Thresholds:

• \>2000 LOC → finding with impact 0.5 ("needs splitting")
• \>1000 LOC → finding with impact 0.3 ("consider splitting")
• \>500 LOC → note only

empirica finding-log --finding "project_commands.py: 4214 LOC — contains 17 unrelated handlers, needs splitting" --impact 0.6

3b. Duplication Detection

Search for functions defined in multiple places:

# Find duplicated function definitions
grep -rn "^def " "$TARGET" | awk -F: '{print $NF}' | sort | uniq -c | sort -rn | head -20
 
# Find copy-pasted patterns (same function name in multiple files)
grep -rn "def get_instance_id\|def find_project_root\|def resolve_project" "$TARGET"

For each duplicate: Read both copies. Assess whether they're identical, similar, or intentionally different. Log accordingly:

# Identical copies
empirica finding-log --finding "get_instance_id() duplicated identically in 6 files" --impact 0.6
 
# Intentionally different (record WHY)
empirica decision-log --choice "Keep separate find_project_root() in hooks" \
  --rationale "Hooks must be standalone — can't import from package" \
  --reversibility exploratory

3c. Module Boundary Assessment

Read the import graph. Check for:

• Circular imports — A imports B, B imports A
• Layer violations — CLI importing from data, data importing from core
• God modules — single file that everything imports

# Check for late imports (circular dependency workaround)
grep -rn "import " "$TARGET" | grep "def \|if " | head -20

3d. Naming & Convention Consistency

Spot check:

• Mixed naming styles (snake_case vs camelCase)
• Inconsistent patterns (some files use handle_X_command, others use X_handler)
• Magic strings/numbers without constants

3e. Error Handling — success-shaped nothing

The count of except: pass is the least of it. The defect that survives lint, types, coverage and a green suite is a failure that returns the value meaning "measured" or "done": return 0 on a COUNT that raised, return [] on a timeout, passed=True on a skipped check, a stub rendering constants as a grade, a verb printing ✅ then ❌ and exiting 0. Lesson b329fdbb1d6ddacb (org pool, lesson-create --from-global) carries the five shapes and a worked day; the probe is:

# 1. Every handler / fallback / default. For EACH: what does the CALLER do with the value?
#    Failure returning the same value as an honest empty (0, [], None, False) with no
#    separate signal is the finding.
rg -n "except .*:\s*$" -A 2 "$TARGET" | rg "return (0|\[\]|\{\}|None|False)|pass$"
 
# 2. Lazily imported modules that do not exist — a verb that has never worked.
#    `# pyright: ignore[reportMissingImports]` on an import is a confession.
#    -N, not -n: a line-number prefix makes every module read MISSING (the probe's own
#    first run did exactly that). Positive control: run it on a file known to import a
#    module that is gone, and see that one name, only.
rg -N "^\s+from (empirica[\w.]*) import" -o -r '$1' "$TARGET" | sort -u \
  | while read m; do python3 -c "import importlib,sys; importlib.import_module(sys.argv[1])" "$m" 2>/dev/null || echo "MISSING $m"; done
git log --diff-filter=D --oneline -- "<path>"   # never existed vs deleted
 
# 3. Stubs rendering numbers: honesty in the docstring, a scorecard on the CLI.
rg -n -i "stub|placeholder|not (yet |fully )?implemented|baseline" "$TARGET"
 
# 4. Flags the parser accepts and nothing reads — an advertised no-op. Builds the
#    live argparse tree and looks for a reader of every dest in the handler modules;
#    tests/test_cli_flags_have_readers.py holds it at zero with a positive control.
python3 scripts/cli_unread_flags.py          # --all adds dests read only via a shared helper

Then run each verb live and read exit code against banner. A parser, a doc page and a passing test do not prove a verb works — investigate had all three and zero working paths for its whole life.

empirica finding-log --finding "<verb> exits 0 with an error line: <path>:<line> returns [] on timeout, caller renders 'no results'" --impact 0.6

When the defect survived a green suite, the test is a suspect. A test that asserts the defective surface (returncode == 0 on the stub) is green exactly where it is blind — rewrite it to the contract and let it go red first.

3f. Matchers — over a formal grammar, or over natural language?

Any check keyed on words or phrases rather than on a grammar. Word lists, antonym or negation sets, risk-word scans, TODO-alikes, "does this text say X" predicates.

The discriminator is FORMAL vs NATURAL, not keyword-matching-good-or-bad. Both are legitimate shapes and both ship in the same repos:

The tell: can I enumerate the tokens and be done? Over a published grammar the spec is finite. Over natural language it never is, and an unfinishable list produces false negatives by construction — so zero hits reads as "nothing to find" when it means "cannot see".

# Word/phrase lists that a checker matches against — audit each one's subject
rg -n "^_?[A-Z_]+(?:WORDS|MARKERS|TERMS|PAIRS|PATTERNS)\s*[:=]" "$TARGET"

For each: what is it matching against? Source code, config, or a wire format → formal, fine. Prose a human or a model wrote → the finding below.

Before condemning one, measure it. Run the predicate over the real corpus it sees, exhaustively where the gate makes that possible, with a positive control proving the instrument is live. A measured zero beats an argument from principle, and it is the difference between "I don't like this" and a finding.

empirica finding-log --finding "Lexical matcher over artifact prose at core/x.py — fired 0 times across N real pairs; instrument live on constructed input" --impact 0.7

The counter-move is not a longer list. Prefer a signal the system already produces from behaviour — a verdict, a test result, an observed outcome — over one that must be inferred by reading text.

Phase 4: Triage

Review all findings. Assign categories and prioritize.

Categories:

Impact scoring guide:

Phase 5: Create Remediation Goals