This project runs a complete A/B test analysis on a landing page redesign, using a synthetic dataset of 10,000 visitors split between Version A (the existing control page) and Version B (a redesigned variant). It answers one business question end to end:
Should we roll out Version B to all traffic?
The full analysis — data generation, statistical testing, segment breakdowns, visualizations, and a final recommendation — lives in ab_testing_analysis.ipynb. This README summarizes the methodology and findings for anyone who wants the results without running the notebook.
10,000 simulated visitor sessions (ab_test_data.csv), generated with a fixed random seed for reproducibility:
| Column | Description |
|---|---|
user_id |
Unique visitor identifier |
group |
A (control) or B (variant) |
time_spent_seconds |
Time spent on the landing page |
pages_viewed |
Number of pages viewed in the session |
clicked_cta |
1 if the user clicked the call-to-action button |
converted |
1 if the user completed the target conversion |
device |
mobile or desktop |
traffic_source |
organic, paid, social, or email |
Group, device, and traffic source are assigned independently of each other (as a real randomized experiment would), so any conversion difference between A and B can be attributed to the landing page rather than to imbalanced traffic mix. Group allocation came out to 5,015 (A) / 4,985 (B) — close to the intended 50/50 split.
- Sample size / power validation — Before trusting the result, we calculate the minimum sample size per group needed to detect the observed effect at 95% confidence / 80% power, and confirm the collected sample clears that bar.
- Conversion rate comparison — Aggregate conversion rate, absolute lift, and relative lift between A and B.
- Chi-square test of independence — Tests whether conversion is statistically independent of landing page version.
- Two-proportion z-test & 95% confidence interval — Quantifies the size of the lift and the range it's likely to fall within.
- Segment analysis — Repeats the conversion comparison and chi-square test within each
deviceandtraffic_sourcevalue, to check the result isn't being driven by one slice of traffic. - Visualizations — Conversion funnel (view → click → convert), conversion rate with 95% CI error bars, segment breakdowns, and engagement metrics (time on page, pages viewed).
| Metric | Value |
|---|---|
| Conversion rate — Version A | 9.57% |
| Conversion rate — Version B | 12.86% |
| Absolute lift (B − A) | +3.29 percentage points |
| Relative lift | +34.4% |
| 95% CI for the lift | [2.05%, 4.52%] |
| Chi-square statistic (df=1) | 27.14, p < 0.000001 |
| Two-proportion z-test | z = 5.22, p < 0.000001 |
| Required sample size per group (80% power) | 1,446 |
| Actual sample size per group | ~5,000 |
The result is statistically significant and adequately powered. The confidence interval for the lift excludes zero, so even under the most conservative plausible read of the data, Version B still wins.
Segment analysis:
- Device — Version B wins on both desktop (+3.69pp, p = 0.0004) and mobile (+3.06pp, p = 0.0001); both are individually significant and of similar magnitude.
- Traffic source — Version B wins directionally on all four channels. Email (+5.24pp, p = 0.0024), organic (+2.44pp, p = 0.0135), and social (+4.92pp, p = 0.0002) are each individually significant. Paid traffic (+2.17pp) is positive but does not reach significance on its own (p = 0.098) — most likely a smaller-sample artifact of slicing an already-significant overall result rather than a real exception, but worth watching post-launch.
- Engagement — Version B users spend more time on the page (137.5s vs. 119.5s) and view more pages per session (4.41 vs. 4.02), reinforcing that the conversion lift reflects genuinely higher engagement rather than a statistical fluke.
Recommendation: Roll out Version B to 100% of traffic.
The lift is real (not noise), adequately powered, meaningful in size (+34% relative), and consistent across devices and traffic sources — there is no segment where B underperforms A. Post-launch, keep an eye on the paid-traffic channel specifically, since it hasn't yet cleared statistical significance on its own, and continue monitoring overall conversion for at least one business cycle to rule out a novelty effect.
pip install numpy pandas matplotlib scipy jupyter
jupyter nbconvert --to notebook --execute --inplace ab_testing_analysis.ipynbAll figures (funnel_chart.png, conversion_rate_ci.png, segment_conversion_rates.png, engagement_metrics.png) and the raw dataset (ab_test_data.csv) are regenerated by running the notebook end to end.