How this instrument works
An A/B test compares two versions of something — a landing page, a checkout flow, an email subject line — by splitting traffic between them and counting conversions. The raw conversion rates almost never match exactly, even when the two variants are genuinely identical, because of ordinary sampling noise. The two-proportion z-test asks a sharper question than 'which rate is higher': it asks how many standard errors apart the two rates are, once you account for how much noise is expected at these sample sizes.
The calculation pools both variants' conversions into a single estimate of the shared conversion rate, then uses that pooled rate to compute a standard error — the expected amount of random wobble between two samples of these sizes if there were truly no difference. Dividing the observed gap between pA and pB by that standard error gives the z-statistic: a small z means the gap is well within what noise alone could produce, while a large z means the gap is bigger than noise easily explains.
This instrument reports the z-statistic and the relative uplift themselves rather than a final significance verdict, the same scope this site's other hypothesis-test calculators keep — comparing the z-statistic against a critical value (1.96 for a common 95% two-tailed threshold) is the next step, and that comparison depends on a significance level you choose, not one this page assumes for you. It also does not correct for peeking at results early or running many variants at once, both of which inflate false-positive rates beyond what a single z-test accounts for.
- Enter the number of people who saw variant A into Visitors — variant A, and how many converted into Conversions — variant A.
- Enter the same two figures for the other variant into Visitors — variant B and Conversions — variant B.
- Read Conversion rate — A and Conversion rate — B for the two raw rates.
- Read Relative uplift, B vs A (%) for how much higher or lower B's rate is, proportionally.
- Read the z-statistic (two-proportion test) and compare its absolute value against a critical value such as 1.96 for 95% confidence to judge significance.
Worked example — 1,000 visitors per variant, 10% vs 12%
A checkout page test sends 1,000 visitors to the current design (variant A) and 1,000 to a redesigned one (variant B). Variant A converts 100 of them; variant B converts 120. Enter those four numbers and Conversion rate — A reads 0.10, Conversion rate — B reads 0.12, and Relative uplift, B vs A (%) reads 20% — variant B is converting a fifth more often than A, in relative terms.
To judge whether that gap is more than noise, the instrument pools both variants: p̂ = (100+120)/(1000+1000) = 0.11. The standard error under that pooled rate works out to about 0.013993, and dividing the 0.02 gap between pB and pA by that standard error gives z ≈ 1.4293. That falls short of the common 1.96 threshold for 95% confidence, so on this sample size alone the 12% vs 10% split isn't yet distinguishable from ordinary sampling noise — a larger sample, a bigger true effect, or both would be needed to push z past that line.
Questions
What counts as a 'statistically significant' result here?
Compare the absolute value of the z-statistic this instrument returns against a critical value for the significance level you've chosen — 1.96 for a common two-tailed 95% threshold, 2.576 for 99%. If |z| exceeds that threshold, the gap between the two conversion rates is larger than the pooled standard error alone would typically produce. This page reports the statistic itself rather than picking a threshold for you, since the right significance level depends on your own tolerance for false positives.
Why pool the two conversion rates before computing the standard error?
The z-test starts from the assumption that both variants share one true underlying conversion rate — the null hypothesis of no difference — so it estimates that shared rate by pooling all conversions and all visitors together, rather than using either variant's rate on its own. That pooled estimate feeds into the standard error, which is what the observed gap between pA and pB gets compared against.
Is a bigger uplift percentage always more significant?
No — uplift and significance answer different questions. A large uplift on a tiny sample can still produce a small z-statistic, because the standard error is wide when there's little data, while a modest uplift on a huge sample can produce a large z-statistic, because the standard error shrinks as sample size grows. Always read Relative uplift, B vs A (%) and the z-statistic together, not either alone.
How many visitors do I need before trusting this test?
There's no single number — it depends on the baseline conversion rate and how small a real difference you want the test to reliably detect, which is a sample-size (power) calculation this instrument doesn't run. As a rule of thumb, low-single-digit conversion rates and modest uplifts need thousands of visitors per variant before the z-statistic reliably clears a typical significance threshold; higher baseline rates need fewer.
Can this calculator handle more than two variants?
No — it's built specifically for a two-proportion comparison, A against B. Testing three or more variants at once with repeated pairwise z-tests inflates the overall false-positive rate beyond what any single test accounts for, so that setup calls for a multiple-comparison correction or an ANOVA-style approach rather than running this test repeatedly.