damianvtran/local-operator/data-analysis

Data analysis

Numbers that survive scrutiny: profile first, keep queries reproducible, carry provenance, sanity-check the result.

Profile before trust

Keep the profile as a small set of queries you can rerun, not ad-hoc typing.

Reproducible queries

• Keep the exact query behind every number in the report. A number nobody can re-derive is a rumour.
• Deterministic: pin the snapshot or time window, make ordering explicit before any limit, and avoid "latest row" ambiguity.
• Work in steps; save intermediates. After changing an input, rerun the step rather than mentally adjusting its output.

Guard silent joins and filters

• Count rows before and after every join. An inner join on an incomplete key drops rows without an error.
• Check filter semantics: inclusive or exclusive bounds, null membership, timezone edges. Apply each filter once.
• State the population: "of the 4,210 rows matching X" beats "of the data".

Provenance and checks

• Every reported number carries its source (table, snapshot), filter, and derivation. Derive it, or say where it came from.
• Reconcile: totals equal the sum of their parts; an independent cut of the same data reproduces the number.
• Units on everything: counts, rates, currency. Percentages name their denominator. Magnitudes are checked against expectation; a 1000x jump is a units or join error until proven otherwise.
• Small n: below your stated threshold, report the count, not a rate. "60% (n=5)" misleads; "3 of 5" does not.

Output

• Separate raw from summary: show the summary and keep the raw one step away so the reader can drill down.
• State assumptions in the report: time window, definitions, exclusions, null handling.

Red flags

• Answering from the first query without looking at the data's shape.
• A join with no row count before and after.
• A percentage with no denominator; a total with no source.
• Dropping rows to "clean" data without recording the rule.
• A distribution treated as stable when it trends over time.