By Mehdi HassanWhat AB test sample size do you need?
The AB test sample size you need is the number of visitors per variant required to detect a given lift with a chosen level of confidence. It's calculated from four inputs: your baseline conversion rate, the minimum detectable effect, the significance level and the statistical power. Change any one of them and the number moves, often a lot.
The figure people usually quote comes from that calculation with common defaults. Spotler's guide to A/B test traffic puts it at roughly 15,000 to 20,000 visitors per variant to detect a 15% lift at a 2% baseline. A standard two-sided calculation at 95% confidence and 80% power lands higher, closer to 37,000 per variant. Either way, it's a lot of traffic for a startup page. This article is part of our guide to A/B testing landing pages, which covers the rest of the process.
What do the inputs to a sample size calculation mean?
The four inputs are the baseline rate, the minimum detectable effect, significance and power. You choose the last three; the first is a fact about your page. Understanding them in plain English makes the calculator's answer much less mysterious.
Input | Plain-English meaning | Common default | Effect on sample size |
|---|---|---|---|
Baseline conversion rate | How often the page converts today | Your real rate, from analytics | Lower baseline needs more visitors |
Minimum detectable effect (MDE) | The smallest relative lift you'd want to catch | 10–30% relative for startups | Smaller MDE needs far more visitors |
Significance level | Chance you'll accept a false winner | 5% (95% confidence) | Stricter level needs more visitors |
Statistical power | Chance you'll catch a real lift of that size | 80% | Higher power needs more visitors |
The MDE is the input people get wrong most. It's not the lift you hope for; it's the smallest lift that would be worth shipping. If a 5% improvement wouldn't change any decision, don't size the test to detect one.
How much traffic do you need for A/B testing at startup scale?
At startup scale, you need roughly thousands of visitors per variant to detect very large lifts and tens of thousands to detect moderate ones. The table below shows visitors per variant, calculated with the standard two-proportion formula at 95% confidence and 80% power. Use it to sense-check, then run your own numbers in a calculator.
Baseline rate | 10% lift | 20% lift | 30% lift | 50% lift |
|---|---|---|---|---|
1% | about 163,000 | about 43,000 | about 20,000 | about 7,800 |
2% | about 81,000 | about 21,000 | about 9,800 | about 3,800 |
3% | about 53,000 | about 14,000 | about 6,500 | about 2,500 |
5% | about 31,000 | about 8,200 | about 3,800 | about 1,500 |
10% | about 15,000 | about 3,800 | about 1,800 | about 700 |
Two patterns stand out. Halving the lift you want to detect roughly quadruples the sample. And doubling the baseline rate roughly halves it. That's why tests on high-intent steps, like a pricing page or a demo form, are often more practical than tests on a cold homepage.
How long should an A/B test run?
An A/B test should run until it reaches the planned sample, and always in whole weeks. To estimate the duration, divide the total sample (visitors per variant times the number of variants) by the weekly traffic the test will actually get.
For example, suppose a SaaS landing page gets 4,000 visitors a week
For example, suppose a SaaS landing page gets 4,000 visitors a week and converts at 3%. The team wants to test a new headline.
To detect a 20% relative lift (3% to 3.6%), the table says about 14,000 visitors per variant, so 28,000 in total. At 4,000 a week, that's seven weeks. Too long for most teams, and long enough for other changes to muddy the result.
To detect a 30% lift (3% to 3.9%), it's about 6,500 per variant, 13,000 in total, a bit over three weeks. That's testable, but only for a change bold enough to plausibly move conversion by 30%, such as a new offer or a rewritten hero, not a button tweak.
This is a hypothetical example to show the arithmetic, not a real result.
If the duration comes out longer than six weeks or so, you have three options: test a bigger change, test on a higher-traffic or higher-converting page, or don't test and improve the page another way. Running a test for three months to detect a small lift usually costs more in delay than it returns.
When should you not A/B test yet?
Don't A/B test yet if you can't reach the required sample for a 30% relative lift within about four to six weeks. Below that line, a formal test will either run for months or be called early on noise, and both waste time you could spend improving the page directly.
- Find your weekly visitors to the page you'd test, from analytics, excluding internal traffic and bots.
- Find your baseline conversion rate for the action you'd test, over the last four to eight weeks.
- Look up the sample for a 30% lift at that baseline, and multiply by two for A and B.
- Divide by weekly visitors. More than six weeks means don't test yet; under four means you have room to test smaller changes.
Below the line, use research-led improvement. Interviews, session recordings and a five-second test of your hero section find the biggest problems without needing statistical significance. Our guide to CRO for low-traffic sites covers the full method, and writing for awareness levels is often the fastest fix for a page that attracts the wrong expectations.
How do you get more out of limited traffic?
Get more out of limited traffic by testing bigger changes, testing closer to the conversion and testing fewer variants at once. Each of those raises the chance of a detectable result without adding a single visitor.
- Test bold changes: a new offer, a different page structure or a rewritten hero, rather than small tweaks.
- Test high-intent pages: pricing, demo request and sign-up pages convert at higher rates, so they need smaller samples.
- Use A/B, not A/B/n: every extra variant needs its own full sample.
- Choose a primary metric that happens often enough, such as starting the form, as long as it genuinely predicts the final conversion.
- Avoid overlapping tests on the same page and audience, which split your traffic further.
Before you launch, make sure the variant is built and tracked properly; a broken test wastes the traffic you worked to collect. Our website QA testing guide covers the checks, and A/B testing mistakes that produce false winners explains what goes wrong in analysis. For what to test once you're past the threshold, see our ranked A/B test ideas.
