🧪

A/B Test Significance Calculator

Compare a conversion rate (two-proportion z-test) or a continuous metric like revenue or time-on-page (Welch's t-test). Get the lift, p-value, confidence, a CI chart, and the null distribution — one- or two-tailed, at any confidence level.

Metric type:

Control (A)

VisitorsConversions

Conversion rate: 8.00%

Variant (B)

VisitorsConversions

Conversion rate: 12.00%

Hypothesis:

Confidence level:

Custom%

Variant (B) wins 🎉

At your 95% confidence level this difference is unlikely to be chance.

p = 0.0029<α = 0.0500✓ clears the bar

Rate A

8.00%

Rate B

12.00%

Relative lift

+50.00%

p-value

0.0029

Confidence

99.7%

z-score

2.98

Conversion rates with confidence intervals

When the whiskers overlap a lot, the gap is probably noise.

The null distribution

Shaded tail = the p-value. Dashed lines = your threshold; they move with the confidence level.

How it’s calculated

It’s a two-proportion z-test. We compare each variant’s conversion rate $\hat p_A = x_A/n_A$ and $\hat p_B = x_B/n_B$ , then ask how many standard errors apart they are under the assumption that they’re really the same:

z = \frac{\hat p_B - \hat p_A}{\sqrt{\hat p\,(1-\hat p)\left(\dfrac{1}{n_A}+\dfrac{1}{n_B}\right)}}, \quad \hat p = \frac{x_A + x_B}{n_A + n_B}

That $z$ becomes a p-value — the chance of seeing a gap this big (or bigger) if the two variants were actually identical. Confidence is just $1 - p$ . When the p-value drops below your threshold (e.g. 0.05 for 95%), the result is “significant.”

Measuring a continuous value instead — revenue per user, time on page — Switch the metric type and it compares the two means with Welch’s t-test, which doesn’t assume the two groups have the same spread:

t = \frac{\bar x_B - \bar x_A}{\sqrt{\dfrac{s_A^2}{n_A} + \dfrac{s_B^2}{n_B}}}

A one-sided hypothesis (“B is better”) puts the whole p-value in one tail; two-sided (“B differs”) splits it across both. Two-sided is the safer default.

← All tools