walrusquant/sports-analytic-skills/sports-modeling-doctrine

Sports Modeling Doctrine

Outcome

Write a modeling charter before acquiring data or fitting models. The charter is a user-owned source of truth for the question, target, timing, evaluation, and shape of done.

Before code, lock the scientific contract:

• what question is being answered;
• what one row represents;
• what is knowable at decision time;
• what baseline counts as evidence of value;
• how success will be measured out of time;
• what result causes acceptance, revision, or stopping.

If these fields are not named, do not fit a model.

When to use this skill

Use it when starting or reframing a sports analysis, deciding what “good” means, reviewing whether a result is evaluable, or preventing exploratory work from quietly becoming a predictive claim.

After the charter, use only the relevant specialist skills: a public-source loader for acquisition, eda-sports for data understanding, feature-rules for time-safe predictors, baseline-models for reference models, and the relevant statistical, predictive, rating, simulation, or validation skill.

Question types

Do not mix explanatory and predictive success. A stable association need not improve forecasts, and a useful predictor need not identify a causal mechanism.

Charter workflow

1. Write the question in one sentence.
2. Name sport, competition, population, era, and exclusions.
3. Define grain: game, team-game, player-game, possession, play, pitch, or other.
4. Classify the analysis type using the table above.
5. For predictive, ranking, or prospective simulation work, define decision

time T and forecast horizon precisely.

1. Define target or estimand, units, label rules, and treatment of ties or missing outcomes.
2. Record base rate, null expectation, or incumbent process.
3. Name at least one naive and one strong simple baseline.
4. Lock the primary metric and its direction before fitting.
5. Choose validation that respects event, season, and availability order.
6. Define data requirements, minimum coverage, and provenance standards.
7. Write acceptance, failure, revision, and stop conditions.
8. Record out-of-scope decisions and unsupported uses.

Grain and decision-time contract

For each proposed field, be able to answer:

row grain: <what one row represents>
entity/event key: <natural key>
decision time T: <timestamp or event boundary>
forecast horizon: <what future interval is predicted>
feature available by T: <yes/no and source timestamp>
target observed after T: <yes/no>
paired/dependent rows: <shared games, players, series, seasons>

Sports data routinely contain doubled team-game rows, postgame summaries, revised injury reports, end-of-season aggregates, and ratings updated after the event. Column names do not prove availability. Define an as-of rule.

Baseline ladder

Lock baseline definitions, training windows, and missing-data behavior before candidate tuning. All models must be scored on identical held-out rows.

Metric defaults

Accuracy is rarely sufficient for probability work. Match the metric to the decision, define its direction, and compute it only on genuinely held-out data.

Validation defaults

Ordered sports events usually require expanding-window or rolling-origin validation. A default season walk-forward design trains on seasons before s and evaluates on season s. When a sport lacks clean seasons, split by event time with explicit gaps or embargoes where labels/features overlap.

Random row shuffles are not the default because they can mix future roster, team-strength, schedule, and feature information into training. Group shared contests together, and account for repeated teams or players when estimating uncertainty.

Acceptance and stopping rules

A predictive candidate is normally worth keeping only if it:

• beats locked baselines on the mean primary metric;
• improves in a meaningful majority of eligible folds or has a justified pooled result;
• passes time-safety and leakage checks;
• avoids catastrophic calibration if probabilities are quoted;
• remains useful under relevant era, team, player, or event slices;
• justifies its complexity and maintenance burden.

For explanatory work, require stable effect direction, honest uncertainty, and bounded claims. For simulation, require input calibration and sensitivity.

Read `references/good_enough.md` when setting the acceptance rule. Stop when the rule is met, when a failure condition invalidates the work, or when further complexity does not earn material held-out value.

Charter schema

question
sport_and_competition
population_and_exclusions
grain_and_natural_key
analysis_type
decision_time_and_horizon
target_or_estimand
base_rate_or_null
naive_and_strong_baselines
primary_metric_and_direction
secondary_metrics
validation_design
data_requirements_and_provenance
uncertainty_plan
acceptance_rule
failure_and_stop_conditions
out_of_scope_and_prohibited_uses
required_artifacts

Worked charters

Pre-game team-win probability

Question: Do venue and shifted form improve P(team wins)?
Sport/population: NFL completed regular-season games, 2018-2024
Grain: team-game; two paired rows per contest
Decision time: scheduled kickoff
Target: won; ties reported separately
Baselines: constant training prevalence; venue-only logistic
Primary metric: held-out log-loss, lower is better
Validation: season walk-forward; contest rows remain together
Acceptance: beat both baselines in mean and at least five of seven folds
Failure: any feature not provably available before kickoff
Out of scope: causal claims and wagering profitability

Player next-game count

Question: Predict a starter's strikeouts in the next start.
Grain: player-game
Decision time: first pitch
Target: strikeouts in that start
Baselines: trailing shifted mean; opponent strikeout-allowed mean
Primary metric: MAE
Validation: date walk-forward with player histories truncated at T
Cold start: separate rule and reporting slice

Read `references/charter_examples.md` for additional compact NFL, NBA, and MLB patterns.

Non-negotiables and integrity rules

1. Time order matters; random event shuffles need explicit justification.
2. Features must be legal at T for prospective claims.
3. Baselines are defined before candidate models.
4. The primary metric is locked before inspecting test folds.
5. Grain changes require explicit aggregation and key validation.
6. Candidate and baseline use identical held-out populations.
7. Failures and non-improvement are recorded as valid results.
8. Strong claims require a leakage audit and honest uncertainty.
9. Public data does not eliminate provenance, licensing, or snapshot duties.
10. Scope does not expand silently after favorable results appear.
11. Use the simplest model that meets the charter.
12. Record decisions and artifacts outside ephemeral chat.

Anti-patterns

• choosing an algorithm before defining the decision;
• feature soup without an availability timestamp;
• calling one hot season “generalization”;
• changing the primary metric after seeing held-out results;
• treating each team-game row as an independent game;
• copying the same assumptions across sports with different schedules or rules;
• continuing to tune after the test window becomes familiar;
• declaring victory because a complex model beats the null but not a strong simple baseline;
• producing probabilities without a calibration plan.

Helper

python <path-to-sports-modeling-doctrine>/scripts/print_charter_template.py \
  --out data/modeling_charter.md

Fill the user-owned template before data acquisition or fitting. The helper does not choose the question, baseline, or acceptance rule.

Operating and handoff rules

• Use only the skills relevant to the charter; every skill must stand alone.
• Pass explicit user-owned artifacts between acquisition, analysis, and reporting steps.
• Keep the charter current when a justified scope decision changes.
• Re-charter rather than quietly changing target, grain, or decision time.
• Report a blocker when timing, grain, provenance, or evaluation cannot be established.

For optional downstream routing, read `references/skill_path.md`. It is a decision aid, not a requirement to invoke every listed skill.

Output contract

Return or save the completed charter with question, type, sport/population, grain/key, decision time, target, baselines, primary metric, validation, uncertainty, acceptance/failure rules, out-of-scope uses, and required artifacts. Do not return an algorithm recommendation without this contract.

Resources

• `references/charter_examples.md` — read for

sport- and target-specific charter examples.

• `references/good_enough.md` — read when defining

acceptance, failure, and stopping rules.

• `references/skill_path.md` — read when choosing

optional specialist handoffs after the charter.

• scripts/print_charter_template.py — portable charter writer.