A/B Test Significance Calculator
Find out whether your test result is real or just noise. Enter visitors and conversions for each variant to get the conversion rates, lift, p-value and confidence, and plan sample sizes for your next test.
Test results
Verdict
Sample size planner
Visitors needed per variant
How to check A/B test significance
- Enter the visitors and conversions for the control (A) and the variant (B).
- Choose your confidence level; 95% is standard.
- Before starting a test, use the sample size planner so you know how long to run it.
Reading your A/B test
The calculator runs a two-proportion z-test. The p-value is the probability of seeing a difference this large if the variants actually perform the same. A p-value below 0.05 is significant at 95% confidence; it does not mean there is a 95% chance B is better, and it says nothing about whether the lift is big enough to matter.
The confidence interval shows the plausible range of the true difference in conversion rate. If it includes zero, the test has not shown a difference. Relative lift is the change in B's rate compared with A's.
Decide the sample size before you start and avoid stopping as soon as the result looks significant; peeking repeatedly inflates false positives. Run tests for full weeks to cover day-of-week patterns, and test one main change at a time. Small baseline rates and small lifts need surprisingly large samples.
Need a different tool? Suggest it here — we read every message.
Frequently asked questions
What does statistically significant mean?
The observed difference would be unlikely (below your chosen threshold) if there were no real difference between the variants.
How many visitors do I need?
It depends on your baseline rate and the lift you want to detect; detecting a 10% lift on a 3% rate needs over 50,000 visitors per variant at 80% power.
One-sided or two-sided?
Two-sided is safer and standard; use one-sided only if you decided in advance you only care whether B is better.
Can I stop the test early?
Stopping when results first look significant increases false positives. Reach the planned sample size first.