Hypothesis Testing in Python: p-values, t-tests and Chi-square

Your new checkout page has a slightly higher average order value. Is that a real improvement or just random variation? Hypothesis testing gives a disciplined way to answer.

The idea#

1. State a null hypothesis H₀: "there is no difference" (or no effect).
2. State an alternative hypothesis H₁: "there is a difference".
3. Compute a p-value: if H₀ were true, how likely would we be to see a result at least as extreme as ours, just by chance?
4. If the p-value is below a significance level α chosen in advance (usually 0.05), reject H₀: the data is unlikely under "no difference". Otherwise, you fail to reject H₀.

A small p-value says "this would be surprising if nothing were going on". It is not the probability that H₀ is true.

One-sample t-test#

A packaging machine should fill bags with 500 g. A sample of 12 bags:

from scipy import stats

weights = [497, 501, 495, 498, 502, 494, 499, 496, 500, 493, 497, 498]
print(round(sum(weights) / len(weights), 2))
result = stats.ttest_1samp(weights, popmean=500)
print(round(result.statistic, 3), round(result.pvalue, 4))

497.5
-3.153 0.0092

The average is under 500 g, and the p-value is well below 0.05: a gap this large would be very unlikely if the machine were really filling 500 g on average. Conclusion: the machine is under-filling and should be checked.

Two-sample t-test#

Do customers who saw a new recommendation widget spend more? Compare two independent groups with ttest_ind:

import numpy as np
from scipy import stats

rng = np.random.default_rng(42)
old = rng.normal(loc=800, scale=150, size=200)
new = rng.normal(loc=830, scale=150, size=200)

print(round(old.mean(), 1), round(new.mean(), 1))
result = stats.ttest_ind(new, old, equal_var=False)
print(round(result.statistic, 3), round(result.pvalue, 4))

795.4 833.0
2.628 0.0089

equal_var=False runs Welch's t-test, which does not assume both groups have the same spread; it is a safe default. Compare the p-value with 0.05 to decide.

Because this data is simulated, we know the truth: the new group's average really was set ₹30 higher. With real data you never know the truth, which is exactly why the test is needed.

Paired t-test#

When the same people are measured twice (before and after training), use the paired test on the differences:

from scipy import stats

before = [62, 70, 55, 80, 66, 74, 59, 68]
after = [68, 75, 60, 82, 70, 79, 63, 71]
result = stats.ttest_rel(after, before)
print(round(sum(a - b for a, b in zip(after, before)) / len(before), 2))
print(round(result.statistic, 3), round(result.pvalue, 5))

4.25
9.379 3e-05

Every person improved a little, so the evidence is very strong even with only 8 people. A paired design removes the differences between people, which makes small effects easier to detect.

Chi-square test for categories#

Is the payment method related to the city? For counts in categories, use a chi-square test of independence on a contingency table:

import pandas as pd
from scipy import stats

table = pd.DataFrame({"card": [120, 90], "upi": [180, 210], "cash": [50, 100]}, index=["Mumbai", "Delhi"])
print(table)
chi2, p, dof, expected = stats.chi2_contingency(table)
print(round(chi2, 2), round(p, 5), dof)
print(pd.DataFrame(expected, index=table.index, columns=table.columns).round(1))

        card  upi  cash
Mumbai   120  180    50
Delhi     90  210   100
20.02 5e-05 2
         card    upi  cash
Mumbai   98.0  182.0  70.0
Delhi   112.0  208.0  80.0

expected shows the counts we would see if payment method and city were unrelated. The observed counts differ from them (Delhi uses much more cash), and the small p-value says the difference is unlikely to be chance.

Confidence intervals#

A p-value answers yes or no; a confidence interval shows how big the effect plausibly is. A 95% confidence interval for a mean:

import numpy as np
from scipy import stats

weights = np.array([497, 501, 495, 498, 502, 494, 499, 496, 500, 493, 497, 498])
mean = weights.mean()
sem = stats.sem(weights)                       # standard error of the mean
low, high = stats.t.interval(0.95, df=len(weights) - 1, loc=mean, scale=sem)
print(round(mean, 2), round(low, 2), round(high, 2))

497.5 495.75 499.25

500 g is outside the interval, consistent with the t-test's conclusion. Report intervals alongside p-values: "the average fill is 497.5 g (95% CI 495.75 to 499.25)" tells the reader much more than "p < 0.05".

Errors and limits#

• With α = 0.05, about 1 in 20 tests of a true null hypothesis will still come out "significant". Running many tests and reporting only the significant ones produces false discoveries.
• With huge samples, tiny and unimportant differences become significant. Statistical significance is not practical importance; look at the size of the effect.
• "Not significant" does not prove there is no difference; the sample may be too small to detect it.

Common mistakes#

Practice#

import numpy as np
from scipy import stats

a = np.array([6, 8, 7, 9, 10, 7, 8, 9, 11, 8])
b = np.array([5, 6, 5, 7, 6, 8, 5, 6, 7, 6])
result = stats.ttest_ind(a, b, equal_var=False)
print(a.mean(), b.mean(), round(result.pvalue, 4))
print("significant" if result.pvalue < 0.05 else "not significant")

8.3 6.1 0.0014
significant

Key takeaway#

Next, hypothesis testing for product decisions: A/B Testing.