sanand0/scripts/data-analysis

Investigative Data Analysis

Hunt for stories that make smart readers lean forward and say _"wait, really?"_ - findings that are high-impact, surprising, actionable, and defensible.

This is a DETAILED process. PLAN and execute step by step.

1. Define What Matters

• Who are the audiences and what are their key questions?
• What decisions could findings actually inform? What's actionable vs. merely interesting?
• What would _contradict conventional wisdom_ or reveal hidden patterns?

2. Understand the Data

• Structure: Dimensions (categorical) vs. measures (numeric), types, granularity, field relationships.
• Quality: Completeness, missing values, outliers, duplicates, encoding issues.
• Distribution: Value ranges, (log) normality, skewness, heavy tails, zero-inflation.
• Derived potential: Computable metrics (features, targets), joins, aggregations, time-series constructions.

3. Hunt for Signal

Apply diverse analysis toolkits ranging from statistical tests to geospatial, network, NLP, time series, cohort, segmentation, survival analysis, etc. to expand the insights pool.

Look for stories that _confirm something suspected but never proven_, or _overturn something everyone assumes is true_:

• Extreme/unexpected distributions: What's at the tails? What shouldn't be there?
• Pattern breaks: Where does a trend suddenly shift? What changed, and when?
• Surprising correlations: What moves together that shouldn't? What's independent that should correlate?
• Standout entities: Who dramatically overperforms or underperforms relative to peers? Who drives trends vs. bucks them?
• Hidden populations: What patterns disappear in aggregate but emerge in subgroups (segments, clusters)? (Watch for Simpson's Paradox.)
• Dot connections: What patterns emerge when combining fields that seem unrelated at first?
• Clusters: What clusters or communities emerge? Where are the overlaps and outliers?

Search internally / externally:

• Discover domain-specific rules, context, that have an impact
• Search for WHY this happened
• Surface confounders
• Explore prior research

Find leverage points:

• Underutilized resources or capabilities
• Phase transitions: thresholds where behavior shifts nonlinearly
• Tipping points: what small change would move the aggregate needle?
• What actions are _specific and implementable_, not just directionally correct?

4. Verify & Stress-Test

Verify without rewarding surprise, restating it neutrally. Interestingness is not evidence.

Cross-check externally: Is there outside evidence (benchmarks, research, industry data) that supports, refines, or contradicts the finding?

Test robustness: Does the finding hold under cross model checks, alternative model specs, thresholds, sub-samples, or time windows? Does a placebo test (shuffled labels, random baseline) reproduce it? If so, it's noise.

Check for errors & bias: Examine data provenance, definitions, collection methodology. Control for confounders, base rates, uncertainty. What's _missing_? Selection and survivorship bias are silent killers.

Check for logical fallacies:

• Correlation vs. causation: is there a plausible mechanism, or just co-movement?
• Poor metric: mean instead of median, absolute instead of percentile, average of averages, ...?
• Goodhart's Law: is the metric gamed? Does measuring it change behavior?
• Simpson's Paradox: does segmentation flip the trend?
• Regression to the mean: are extreme values just natural variation reverting?
• Occam's Razor: is there a simpler explanation you're overlooking?
• Survivorship/selection bias: what's missing from the data entirely?
• Second-order effects: what happens downstream beyond the immediate impact?
• Inversion: try to _disprove_ the finding. If you can't, it's more credible.

Check alternative explanations using multiple-testing/selection effects where relevant, and whether the conclusion survives reasonable specifications.

Consider limitations: What _cannot_ be concluded? What caveats must accompany the finding to avoid misuse?

5. Prioritize & Package

Select insights that are

• high-impact (meaningful effect sizes vs. base rates, not incremental),
• actionable (specific and implementable, not just "invest more in X"),
• surprising (challenges assumptions, reveals hidden patterns), and
• defensible (robust under scrutiny, bias-checked).

Lead with the most compelling finding -> evidence -> caveats -> what to _do_ with it.

Tone: Write like a journalist, not a statistician. Say "Sales reps in the Northeast close 2x faster, but only for deals under $10K", not "Closure varies by region." Findings should make a smart reader lean forward.