walrusquant/sports-analytic-skills/leakage-audit

Leakage Audit

Outcome and stance

Assume leakage until the pipeline proves time-safe. Produce an adversarial, evidence-backed verdict: CLEAN, REVIEW REQUIRED, or NOT CLEAN, with contaminated fields, affected claims, required repairs, and retest conditions. A strong metric is never evidence of cleanliness.

When to Use This Skill

Use when:

• trusting a backtest or “good” metric for the first time;
• reviewing a feature table before modeling or after a suspiciously strong result;
• auditing target aliases, joins, shifts, preprocessing, or split design;
• the user says the model looks too good / results are unexpected.

Do not use this skill as a substitute for:

• designing legal features from scratch → feature-rules;
• designing folds and metrics → validation-design;
• first-pass panel EDA → eda-sports.

Lock the audit contract

Before inspecting correlations or metrics, write the target, row grain, exact prediction decision time T, eligible population, source publication/revision policy, and intended validation split. If T is ambiguous, the audit cannot pass.

Use the audit checklist during every audit. Consult the leakage-pattern catalog while tracing failure modes and the case studies when performance or a join looks implausible.

Audit sequence

1. Trace the target to its source and identify aliases, transformations, proxies,

and duplicated labels.

1. Inventory every candidate feature, raw source, event/publication/revision time,

and availability at T.

1. Inspect temporal transforms for stable sort and shift-before-roll/expand.
2. Trace opponent, roster, injury, rating, weather, price, and schedule joins.
3. Verify preprocessing, selection, encoding, and tuning fit only on training rows.
4. Verify validation moves forward in time and related rows remain together.
5. Run automated matrix heuristics, then manually investigate every finding.
6. Reproduce suspicious rows and performance jumps from raw inputs.
7. Issue the verdict and block unsupported claims until repair and rerun.

Leakage catalog

Manual review

Target and time

• Is the target post-T by definition, and are canceled/tied/incomplete events handled?
• Are prediction timestamps stored or reconstructable rather than assumed?
• Which provider fields are revised, and which historical vintage is used?
• Could a feature name be harmless while its source table is post-event?

Transformations

ordered = df.sort_values(["team", "event_time", "game_id"])
expected = ordered.groupby("team")["won"].transform(
    lambda s: s.shift(1).expanding(min_periods=1).mean()
)
assert expected.equals(ordered["pre_win_pct"])

Check sort tie-breaks, reset boundaries, offseason carryover, minimum periods, and whether the first entity event has an unexplained empirical prior. A future perturbation test should prove that changing later outcomes cannot alter earlier features.

Joins

For every join, document left/right grain, keys, expected cardinality, publication-time predicate, unmatched rate, duplicate rate, and fields retained. An as-of join must be backward-looking and satisfy right_published_at <= T. Display-name joins and silent many-to-many expansion are audit failures until resolved. Opponent features must equal the opponent's own pre-event row.

Splits and preprocessing

For every fold prove train max time < test min time, allowing only the declared gap/embargo. Keep paired team-game rows, repeated observations from the same event, and other dependency groups together when the estimand requires it. Imputers, scalers, encoders, selectors, target encoders, PCA, early stopping, and hyperparameter search must see training data only.

Too-good triggers

Near-perfect accuracy, implausibly low log-loss/error, a dramatic single-feature jump, train/test equality, or a result far beyond domain baselines triggers a stop. Recheck target aliases, current-event fields, shifts, opponent joins, duplicates, global preprocessing, and test-fold tuning before discussing merit.

Automated helpers

The audit helper reads a user-owned CSV, Parquet, JSON, JSONL, or NDJSON table, requires pandas, validates named columns, and runs name, exact-target, duplicate, and near-perfect-correlation heuristics.

python /path/to/leakage-audit/scripts/audit_pregame_features.py \
  --input modeling_table.csv --target won \
  --features pre_win_rate,rest_diff,rating_diff \
  --entity-col team --time-col event_time --out leakage.json
 
python /path/to/leakage-audit/scripts/write_audit_stub.py \
  --out data/leakage_audit.md

Use --banned to provide exact forbidden names for the task. The first-event null-rate check is always REVIEW, not PASS, because legitimate priors can populate first events. Duplicate rows and near-perfect correlations are also REVIEW: they are suspicious, not proof by themselves. A direct forbidden or target-identical feature returns NOT CLEAN and status 2. Otherwise the helper returns REVIEW REQUIRED and status 1: heuristics alone never return CLEAN. Only a completed manual audit of lineage, transforms, joins, splits, and fold-local preprocessing can issue the CLEAN verdict.

Exit codes (not crashes)

Stderr also prints exit=1 REVIEW REQUIRED... or exit=2 NOT CLEAN... so a cold agent does not mistake a review signal for a broken script.

Automated checks cannot prove source availability, transform correctness, join direction, or fold scope. They supplement—not replace—lineage and code review.

Diagnosis examples

Rolling form without a shift: manually recomputation shows row i contains its own result. Mark NOT CLEAN, rebuild with shift(1), regenerate all downstream artifacts, and rerun validation.

Final-season efficiency on early games: the value is constant within season and reflects later games. Replace with an expanding as-of statistic from prior events; do not merely drop the first weeks.

Legitimate first-event prior: first rows have non-null ratings, but lineage shows a fixed preseason rating published before T. Keep REVIEW REQUIRED until the initialization and timestamp are documented, then pass that item.

Random K-fold with clean features: feature checks pass but future seasons enter training. The pipeline is still NOT CLEAN; redesign folds and rerun all reported metrics.

Verdict rules

Uncertainty is not CLEAN. A fixed matrix must be regenerated and reevaluated; deleting a suspicious column after scoring does not rescue prior metrics.

Audit report template

Leakage audit
Sport / question / target / grain:
Decision time T and publication policy:
Data snapshot and feature count:
Validation and grouping rule:
Automated helper result / exit status:
Lineage and transform evidence reviewed:
Findings:
  1. [PASS | REVIEW | FAIL] item, evidence, affected scope
Contaminated fields/artifacts:
Required repairs and retests:
Remaining uncertainty:
Verdict: CLEAN | REVIEW REQUIRED | NOT CLEAN
Auditor / date:

Integrity and resource routing

1. Prefer a false review to a false clean; never infer cleanliness from metrics.
2. Timeline sort in the audit must match construction exactly.
3. Do not repair leakage by dropping a bad test period or redefining success.
4. NOT CLEAN blocks predictive claims until rebuilt and retested.
5. Use references/audit_checklist.md for completion,

references/leakage_patterns.md for taxonomy, and references/case_studies.md for diagnosis.

1. Pair with feature-rules for remediation and validation-design for folds.