A/B Testing in Python: Conversion Rates, Significance and Sample Size

An A/B test is a controlled experiment: randomly show some users version A (the current page, the control) and others version B (the change), then compare an outcome such as the conversion rate (the share who buy, sign up or click). Because users are assigned at random, a real difference in outcomes can be attributed to the change itself, which is what correlation alone cannot do.

Designing the test#

Decide these before starting:

1. One change and one primary metric (for example, purchase conversion rate).
2. Random assignment of users (not visits) to A or B, usually 50/50.
3. The minimum effect worth detecting (say, 4.0% → 4.5%) and the sample size needed for it (below).
4. The significance level α (usually 0.05) and how long the test will run.

Then run the test to the planned size, and only then analyse.

Analysing the results#

The test ran with these results:

visitors_a, conversions_a = 10_000, 400
visitors_b, conversions_b = 10_000, 460

rate_a = conversions_a / visitors_a
rate_b = conversions_b / visitors_b
print(f"A: {rate_a:.2%}  B: {rate_b:.2%}")
print(f"absolute uplift: {rate_b - rate_a:.2%} points, relative uplift: {(rate_b - rate_a) / rate_a:.1%}")

A: 4.00%  B: 4.60%
absolute uplift: 0.60% points, relative uplift: 15.0%

B converts better in this sample. Is the difference bigger than chance would produce?

The two-proportion z-test#

For conversion rates, the standard test compares the two proportions. It uses the pooled rate (both groups together) to estimate how much the difference would vary if A and B were really the same:

import math
from scipy.stats import norm

visitors_a, conversions_a = 10_000, 400
visitors_b, conversions_b = 10_000, 460

p_a = conversions_a / visitors_a
p_b = conversions_b / visitors_b
p_pool = (conversions_a + conversions_b) / (visitors_a + visitors_b)
se = math.sqrt(p_pool * (1 - p_pool) * (1 / visitors_a + 1 / visitors_b))
z = (p_b - p_a) / se
p_value = 2 * (1 - norm.cdf(abs(z)))         # two-sided
print(round(z, 3), round(p_value, 4))

2.091 0.0365

The p-value is below 0.05, so at α = 0.05 we reject "no difference": B's higher conversion rate is unlikely to be chance alone. (The statsmodels library has the same test as proportions_ztest; the formula above shows what it computes.)

How big is the improvement?#

A confidence interval for the difference in rates says how large the uplift plausibly is:

import math
from scipy.stats import norm

visitors_a, conversions_a = 10_000, 400
visitors_b, conversions_b = 10_000, 460
p_a, p_b = conversions_a / visitors_a, conversions_b / visitors_b

se_diff = math.sqrt(p_a * (1 - p_a) / visitors_a + p_b * (1 - p_b) / visitors_b)
z = norm.ppf(0.975)
diff = p_b - p_a
print(f"uplift {diff:.2%} points, 95% CI {diff - z * se_diff:.2%} to {diff + z * se_diff:.2%}")

uplift 0.60% points, 95% CI 0.04% to 1.16%

The interval excludes 0 (matching the significant test), but its lower end is small. Whether a change is worth shipping depends on that range, the cost of the change and the business value of each extra conversion.

How many users do you need?#

Small tests miss real improvements. The sample size per group needed to detect a change from rate p₁ to p₂ with significance α and power 1 − β (the chance of detecting a real effect; 80% is common) can be estimated with a standard formula:

import math
from scipy.stats import norm

def sample_size_per_group(p1, p2, alpha=0.05, power=0.8):
    z_alpha = norm.ppf(1 - alpha / 2)
    z_beta = norm.ppf(power)
    p_bar = (p1 + p2) / 2
    numerator = (z_alpha * math.sqrt(2 * p_bar * (1 - p_bar))
                 + z_beta * math.sqrt(p1 * (1 - p1) + p2 * (1 - p2))) ** 2
    return math.ceil(numerator / (p2 - p1) ** 2)

print(sample_size_per_group(0.04, 0.045))   # detect 4.0% -> 4.5%
print(sample_size_per_group(0.04, 0.05))    # detect 4.0% -> 5.0%
print(sample_size_per_group(0.04, 0.06))    # detect 4.0% -> 6.0%

25551
6745
1863

Halving the effect you want to detect roughly quadruples the users you need. Small improvements to low conversion rates need large tests, which is why many sites only test bold changes.

Simulating many A/B tests#

A simulation shows why rules about sample size and stopping matter. Here, A and B are identical (both 4%), yet some tests still come out "significant":

import numpy as np
from scipy.stats import norm

rng = np.random.default_rng(3)
n, rate, tests = 5_000, 0.04, 2_000
a = rng.binomial(n, rate, size=tests)
b = rng.binomial(n, rate, size=tests)
p_pool = (a + b) / (2 * n)
se = np.sqrt(p_pool * (1 - p_pool) * (2 / n))
z = (b - a) / n / se
p_values = 2 * (1 - norm.cdf(np.abs(z)))
print(f"{(p_values < 0.05).mean():.1%} of tests between identical versions were 'significant'")

5.4% of tests between identical versions were 'significant'

About 5% of tests are false positives: exactly what α = 0.05 means. Peeking at results every day and stopping as soon as p < 0.05 raises this rate much higher.

Common pitfalls#

• Peeking and stopping early: decide the sample size first, and analyse at the end.
• Testing many metrics and reporting whichever is significant: pick one primary metric.
• Novelty effects: users react to anything new; run tests for at least one full business cycle (often a week or two).
• Mismatched groups: if the A/B split is not close to the planned 50/50, something is wrong with the assignment; check before trusting results.
• Practical significance: a "significant" 0.05-point uplift may not be worth the engineering cost.

Practice#

import math
from scipy.stats import norm

na, ca, nb, cb = 8_000, 640, 8_000, 700
pa, pb = ca / na, cb / nb
pool = (ca + cb) / (na + nb)
se = math.sqrt(pool * (1 - pool) * (1 / na + 1 / nb))
z = (pb - pa) / se
p = 2 * (1 - norm.cdf(abs(z)))
print(f"{pa:.2%} {pb:.2%}", round(z, 3), round(p, 4))
print("significant" if p < 0.05 else "not significant")

8.00% 8.75% 1.712 0.0868
not significant

Key takeaway#

Next, showing your findings: matplotlib Basics.