A/B Test Significance & Lift Calculator

About the A/B Test Significance

This calculator checks whether the gap in conversion rate between two A/B test variants is a genuine effect or something that could plausibly be random noise. Instead of just comparing raw percentages, it runs a two-proportion z-test on the visitor and conversion counts you enter for Variant A and Variant B. It is built for reading results after a test has already run, not for deciding how many visitors you need before starting one.

How It Works

You enter visitor and conversion counts for each variant, and the calculator first works out each variant's conversion rate and the relative lift of B over A. It then pools both variants' conversions and visitors into a single combined proportion, uses that to estimate the standard error of the difference between the two rates, and divides the observed difference by that standard error to get a z-score. If the absolute value of that z-score is 1.96 or higher, the result is flagged as statistically significant at the 95% confidence level.

rateA = conversionsA / visitorsA; rateB = conversionsB / visitorsB; relative lift = (rateB - rateA) / rateA x 100; pooled p = (conversionsA + conversionsB) / (visitorsA + visitorsB); standard error = sqrt(pooled p x (1 - pooled p) x (1/visitorsA + 1/visitorsB)); z = (rateB - rateA) / standard error; significant if |z| >= 1.96.

Formula & Methodology

To reproduce this by hand, start with each variant's raw conversion rate, then find the pooled conversion rate across both variants combined, since the test assumes both variants share one true underlying rate. Plug that pooled rate into the standard error formula, which factors in both variants' sample sizes separately. The z-score is the actual difference in rates divided by that standard error, and comparing its absolute value against 1.96 corresponds to a two-tailed test at the 95% confidence level, the threshold this calculator uses.

Examples

A large-sample test with a real winner

Variant A converts 250 of 5,000 visitors (5.0%) and Variant B converts 300 of 5,000 visitors (6.0%), a 20% relative lift. The pooled conversion rate is 5.5%, giving a standard error near 0.00456 and a z-score around 2.19, which clears the 1.96 threshold, so the result is statistically significant.

A bigger lift that isn't significant yet

Variant A converts 20 of 200 visitors (10%) and Variant B converts 28 of 200 visitors (14%), a larger 40% relative lift. But with only 200 visitors per variant, the standard error is wider and the z-score works out to roughly 1.23, below 1.96, so the calculator marks this result as not statistically significant despite the bigger apparent lift.

Advantages

Turns two sets of raw counts into a single, standardized read on whether a difference is likely real, instead of relying on eyeballing percentages.
Surfaces relative lift and the underlying z-score side by side, so you can see both the size of the effect and the statistical confidence behind it.
Works instantly on results you already have, without needing a statistics background or a separate spreadsheet formula.

Common Mistakes

Treating any positive lift as proof the new variant won, without checking whether the z-score actually clears the significance threshold.
Running the calculator on a test that was stopped early because the numbers 'looked good,' which raises the chance of a false positive compared to a test that ran its planned duration.
Comparing variants with very different visitor counts without noticing that the imbalance widens the standard error and makes true differences harder to detect.

Edge Cases to Watch For

If either variant has zero visitors, the calculator returns an error instead of dividing by zero.
When Variant A has zero conversions, relative lift is reported as 0% rather than an undefined value, since there is no baseline rate to compare against.
If the pooled conversion rate works out to exactly 0% or 100%, the standard error becomes zero and the z-score defaults to 0, which will show as 'not significant' even if the raw rates differ - a sign the sample is too extreme or too small to draw conclusions from.

Common Use Cases

Growth and marketing teams reading out the results of a landing page, email subject line, or checkout flow test.
Product managers deciding whether a feature experiment moved a key conversion metric enough to justify a full rollout.
Analysts double-checking a colleague's 'we found a winner' claim before it gets reported up as fact.

Frequently asked questions

How is this different from an A/B test sample size calculator?

A sample size calculator tells you how many visitors you need BEFORE running a test, based on your baseline rate and the minimum effect you want to detect. This calculator evaluates results AFTER the test has run, using a two-proportion z-test to check whether the observed difference between variants is likely real or just random noise.

Conclusion

A statistically significant result at the 95% level means the observed gap is unlikely to be pure chance, but it does not guarantee the effect will hold at the same size going forward. Pairing this calculator with a sensible test duration and sample size, and treating a 'not significant' result as inconclusive rather than as proof the variants are equal, keeps the interpretation honest.