Data Analysis Litiao

When to Load#

User asks about: analyzing data, finding patterns, understanding metrics, testing hypotheses, cohort analysis, A/B testing, churn analysis, statistical significance.

Core Principle#

Analysis without a decision is just arithmetic. Always clarify: What would change if this analysis shows X vs Y?

Methodology First#

Before touching data:

1. What decision is this analysis supporting?
2. What would change your mind? (the real question)
3. What data do you actually have vs what you wish you had?
4. What timeframe is relevant?

Statistical Rigor Checklist#

• Sample size sufficient? (small N = wide confidence intervals)
• Comparison groups fair? (same time period, similar conditions)
• Multiple comparisons? (20 tests = 1 "significant" by chance)
• Effect size meaningful? (statistically significant ≠ practically important)
• Uncertainty quantified? ("12-18% lift" not just "15% lift")

Analytical Pitfalls to Catch#

PitfallWhat it looks likeHow to avoidSimpson's ParadoxTrend reverses when you segmentAlways check by key dimensionsSurvivorship biasOnly analyzing current usersInclude churned/failed in datasetComparing unequal periodsFeb (28d) vs March (31d)Normalize to per-day or same-length windowsp-hackingTesting until something is "significant"Pre-register hypotheses or adjust for multiple comparisonsCorrelation in time seriesBoth went up = "related"Check if controlling for time removes relationshipAggregating percentagesAveraging percentages directlyRe-calculate from underlying totals

For detailed examples of each pitfall, see pitfalls.md.

Approach Selection#

Question typeApproachKey output"Is X different from Y?"Hypothesis testp-value + effect size + CI"What predicts Z?"Regression/correlationCoefficients + R² + residual check"How do users behave over time?"Cohort analysisRetention curves by cohort"Are these groups different?"SegmentationProfiles + statistical comparison"What's unusual?"Anomaly detectionFlagged points + context

For technique details and when to use each, see techniques.md.

Output Standards#

1. Lead with the insight, not the methodology
2. Quantify uncertainty — ranges, not point estimates
3. State limitations — what this analysis can't tell you
4. Recommend next steps — what would strengthen the conclusion

Red Flags to Escalate#

• User wants to "prove" a predetermined conclusion
• Sample size too small for reliable inference
• Data quality issues that invalidate analysis
• Confounders that can't be controlled for