A/B Test Significance Calculator

Enter the visitors and conversions for two variants to compare conversion rates, measure the relative uplift, and test whether the difference is statistically significant with a two-proportion z-test.

Quick Facts

Method
Two-proportion z-test (pooled), two-tailed
z = (p̂B − p̂A) / √[p̂(1−p̂)(1/nA + 1/nB)], where p̂ is the pooled rate.

Your Results

Calculated
Conversion rates
-
Variant A vs Variant B
Relative uplift (B vs A)
-
Change in conversion rate
z-statistic
-
Pooled two-proportion test
p-value (two-tailed)
-
Probability under the null

Ready

Enter visitors and conversions for both variants, then run the test.

What this A/B test calculator does

An A/B test (or split test) shows two versions of something — a landing page, a headline, a checkout button — to two randomly assigned groups of users, then measures which version produces a higher conversion rate. This calculator takes the raw counts from both groups (how many visitors each variant received and how many converted) and answers the key question: is the difference in conversion rates real, or could it easily be the result of random chance?

It does this with a two-proportion z-test, the standard significance test for comparing two success rates. You get each variant's conversion rate, the relative uplift of B over A, the z-statistic, and a two-tailed p-value, plus a plain-English verdict at your chosen confidence level.

The formula

Let variant A have nA visitors and cA conversions, and variant B have nB visitors and cB conversions. The observed conversion rates are:

  • p̂A = cA / nA  and  p̂B = cB / nB

Under the null hypothesis both variants share one true rate, estimated by the pooled proportion p̂ = (cA + cB) / (nA + nB). The pooled standard error of the difference is:

  • SE = √[ p̂ (1 − p̂) (1/nA + 1/nB) ]

The test statistic and p-value are:

  • z = (p̂B − p̂A) / SE
  • p-value = 2 × [ 1 − Φ(|z|) ], where Φ is the standard normal CDF (two-tailed test)

If the p-value is less than or equal to your significance level α (0.05 for 95% confidence), the difference is called statistically significant. The relative uplift reported is (p̂B − p̂A) / p̂A, expressed as a percentage.

A worked example

Suppose variant A converts 100 of 1,000 visitors (10.0%) and variant B converts 130 of 1,000 (13.0%). The pooled rate is 230/2000 = 0.115, so SE = √[0.115 × 0.885 × (1/1000 + 1/1000)] ≈ 0.01427. The z-score is (0.13 − 0.10) / 0.01427 ≈ 2.10, giving a two-tailed p-value of about 0.036. Because 0.036 < 0.05, variant B's 30% relative lift is statistically significant at 95% confidence.

Common significance thresholds

  • p ≤ 0.05 — 95% confidence, the default and most widely used threshold.
  • p ≤ 0.01 — 99% confidence, used when a false positive would be costly.
  • p ≤ 0.10 — 90% confidence, sometimes used for early exploratory reads.

Frequently Asked Questions

What does a p-value below 0.05 actually mean?
It means that if both variants truly had the same conversion rate, you would observe a difference at least this large only about 5% of the time by pure chance. It is evidence against the "no difference" hypothesis, not proof that B is better, and it says nothing about how big the improvement is — always read the uplift alongside it.
Why does the calculator use a pooled standard error?
The significance test asks how likely the observed gap is under the null hypothesis that both variants share one true conversion rate. Pooling the conversions from both groups gives the best single estimate of that shared rate, which is why the pooled proportion is used in the standard error. This is the classic two-proportion z-test taught in statistics courses.
How many visitors do I need per variant?
It depends on your baseline rate and the smallest lift you care to detect. As a rough guide, detecting a change from 10% to 13% at 95% confidence with 80% power needs roughly 1,700–2,000 visitors per variant. Smaller effects need dramatically larger samples. Decide your target sample size before the test and avoid peeking early, which inflates false positives.
Is this a one-tailed or two-tailed test?
This calculator runs a two-tailed test, which asks whether the variants differ in either direction. That is the safe default for A/B testing because you usually cannot rule out in advance that a change might hurt conversions. A one-tailed test would report smaller p-values but only detects a change in the direction you pre-specified.

Related calculators