walrusquant/sports-analytic-skills/data-sources

Data Sources

When to Use This Skill

Use when:

• the user needs a public sports data source and is not sure which one;
• choosing grain (game, team-game, player-game, play/event) before loading;
• comparing nflverse / sportsdataverse / pybaseball / other public options;
• writing a source plan with provenance, snapshot, and first-load checks.

Do not use this skill when:

• the user already has a usable table and wants EDA or modeling;
• they explicitly want the optional repository toolkit panels/CLI → sports-ds-bridge;
• they already chose a loader and just need that loader skill.

Outcome

Produce a source plan naming the required grain, decision-time fields, primary source, fallback, coverage risks, terms, snapshot rule, and first-load checks. Source choice does not create predictive validity; it determines whether the question can be answered reproducibly and legally.

Use this skill before acquisition code or whenever coverage, field semantics, historical depth, IDs, latency, or licensing are uncertain. If the source is already chosen and a loader fails, use its package skill instead.

Required inputs

• sport, competition, question, target, and analytical population;
• analytical grain and natural key: season, game, team-game, player-game,

possession, play, pitch, or event;

• each candidate source's native grain and the aggregation/join contract needed

to reach the analytical grain;

• historical depth, expected volume, refresh cadence, and completion latency;
• exact prediction decision time T and fields that must be known by then;
• stable identifiers and cross-source joins required;
• license, attribution, redistribution, authentication, and rate constraints;
• required raw and derived artifact formats.

Candidate guide

These are ecosystems to investigate, not durable rankings. Confirm current coverage, access, and terms in upstream documentation at selection time.

Read the source matrix while shortlisting ecosystems, the terms notes before acquisition or redistribution, and the sanity checks before accepting the first pull.

Choose a source whose native records contain the facts required for the analytical grain, and write the aggregation contract when the grains differ. For example, play data may be aggregated to team-game only after event ordering, completion, and key rules are explicit. A season aggregate cannot manufacture play-level detail or historical publication vintages it never contained.

Source comparison rubric

Compare at least two plausible sources on:

Do not treat “popular” or “easy to import” as sufficient evidence.

Workflow

1. Lock question, analytical grain/key, target, population, and T.
2. List only the minimum required raw fields and stable identifiers.
3. Shortlist and compare a primary and fallback source with the rubric.
4. Read upstream documentation and record field/time semantics and terms.
5. Estimate request count and data volume; bound event-level pulls by dates/IDs.
6. Pull the smallest representative sample spanning relevant eras/statuses.
7. Validate keys, row counts, periods, outcomes, entities, timestamps, and nulls.
8. Test required joins on a small sample and quantify unmatched/many-to-many rows.
9. Save the untouched raw response or immutable snapshot with retrieval metadata.
10. Document fallback triggers, known gaps, and the EDA handoff.

First-load diagnostics

• nonzero rows and requested competitions/seasons are present;
• natural keys are unique at the claimed grain;
• scheduled, live, completed, canceled, and postponed records are distinguishable;
• completed events have plausible outcomes and nonconstant score distributions;
• team-game overall win rate is near its paired identity and home-only rates are plausible;
• identifiers remain stable or have a documented crosswalk;
• timestamps have known timezone and meaning;
• critical missingness is measured by season/status/source;
• source/version/query/retrieval time accompany every saved artifact.

key = ["game_id", "team"]
assert len(df) > 0
print(df["season"].value_counts().sort_index())
print("duplicate key rows:", df.duplicated(key).sum())
print(df.groupby("status")["game_id"].nunique())
print(df.isna().mean().sort_values(ascending=False).head(20))

Reject corrupt or mis-grained pulls rather than adapting the question silently.

Snapshot and provenance policy

Preserve raw responses before normalization. Record source/project version, endpoint or release, exact parameters, requested/retrieved times in UTC, schema fingerprint, row count, checksum, timezone, license/attribution, and any later correction policy. Derived tables should point to immutable raw snapshots and transformation code/config. Never overwrite data referenced by an experiment.

For predictive work, a current bulk snapshot may contain revisions unavailable historically. State whether the study reconstructs real-time vintages or accepts revised-history data and limit claims accordingly.

Hard constraints and anti-patterns

Respect terms, rate limits, access controls, and attribution. Never evade a provider restriction. Cache only when permitted.

Worked source plans

NFL pre-kickoff win model: team-game grain; nflverse schedules/results via nflreadpy; require stable game/team IDs, scheduled start, home/away, completion state, and scores for labels. Check schema across requested seasons and snapshot the release. Use a documented public fallback only if coverage fails.

MLB pitch analysis: pitch grain; pybaseball Statcast; bound every request by date and test a small window. Verify game_pk, at-bat/pitch order, units, missing tracking fields, doubleheaders, and revision behavior before scaling the pull.

Cross-source injury enrichment: select sources with stable player/team IDs and publication timestamps. Build a reviewed crosswalk, measure unmatched joins, and require an as-of predicate. If historical timestamps are absent, the source cannot support the intended pre-event claim.

Source-plan artifact

Question / sport / competition / target:
Analytical grain and natural key / population / decision time T:
Required fields and IDs:
Primary source and rationale:
Primary native grain/key and aggregation contract:
Fallback and trigger:
Fallback native grain/key and compatibility with primary:
Coverage window / status handling:
Event, publication, revision semantics:
License, terms, attribution, access limits:
Estimated request/volume and bounded query:
Representative sample checks:
Join checks and crosswalk:
Raw snapshot location / checksum / retrieved_at:
Known gaps and claim limitations:
EDA handoff:

Create the portable stub with:

python /path/to/data-sources/scripts/print_source_plan.py --out data/source_plan.md

Integrity and resource routing

1. Never claim coverage until a representative sample is inspected.
2. Preserve raw snapshots and provenance for every reproducible claim.
3. Source timestamps and revision policy are part of feature legality.
4. Read source_matrix.md for ecosystem choice, tos_notes.md for access and

redistribution, and sanity_checks.md before accepting data.

1. Hand off loaded data to eda-sports, then feature-rules and validation.