A/B Test Significance Calculator

Check whether an A/B test result is statistically significant: conversion rates, lift, confidence and p-value.

How do I know if my A/B test is significant?

An A/B test is statistically significant when the difference between variants is unlikely to be random. With a two-proportion z-test, a p-value below 0.05 means significance at 95 % confidence.

Example

A: 200 conversions from 5,000 visitors (4 %). B: 250 from 5,000 (5 %). Relative lift +25 %, p ≈ 0.016 — significant at 95 % confidence.

Avoid common mistakes

Decide the sample size before the test (use the Sample Size Calculator) and do not stop early when results look good ("peeking").
Run tests for full weeks to cover weekday and weekend behaviour.
Test one main change at a time.

Frequently asked questions

What does the p-value mean?

The probability of seeing a difference at least this large if the variants were actually the same.

Is 90% confidence enough?

It can be for low-risk changes; 95 % is the common standard.

Is this one-tailed or two-tailed?

Two-tailed, which is more conservative.