How much traffic does an A/B test really need?

What is an A/B test — and why is it so data-hungry?

In an A/B test, a page's traffic is randomly split between two variants: A is the original, B contains the change. You measure which variant produces more conversions — purchases, sign-ups, enquiries. The method comes from clinical research and is, at its core, a statistical experiment.

And that is exactly the problem: a statistical experiment needs a minimum sample to tell effect from chance. Conversions are rare events (typically 2 to 5 out of 100 visitors convert), and the differences between two variants are usually small. The rarer the event and the smaller the expected difference, the more visitors the test needs. This number can be calculated precisely before the test — it is not a matter of opinion but of maths.

The maths: this is how many visitors your A/B test needs

Two figures determine the sample size: the significance level (usually 95% — how sure you want to be that a measured difference is not chance) and the statistical power (usually 80% — the chance of detecting a real difference at all). With these standard values you get:

Calculated with the standard formula for comparing two proportions (95% significance, 80% power, two-sided). Verifiable with the freely available sample size calculator by Evan Miller. “Visitors” means visitors who actually see the tested page — not your website's total traffic.

An A/B test at a 3% conversion rate needs around 28,000 visitors to cleanly prove a 20% improvement. Proving +10% takes over 100,000.Two-proportion test, 95% significance / 80% power — standard test statistics methodology

To put that in perspective: a website with 5,000 visitors per month would need about half a year for the 28,000-visitor test — and almost two years for the 106,000 one. This is where statistics collides with reality: product range, prices, seasonality and ad campaigns change faster than the test runs.

“Then I'll just let the test run longer” — unfortunately not

The obvious workaround fails for two reasons.

First: long tests no longer measure the same thing. Over months, holiday seasons, promotions and campaigns blend into both variants. Returning visitors delete cookies and end up in A one day and B the next. The result gets blurrier, not sharper.

Second — and more insidious: peeking. Almost every testing tool shows a live interim result with a “significance” indicator. If you stop as soon as it turns green, you break the statistics entirely. Statistician Evan Miller ran the numbers back in 2010: if you continuously check a running experiment and stop at the first spike, your real error rate is up to 26% instead of the displayed 5%. Data scientist Martin Goodson reached the same conclusion in simulations in 2014 — with an even more uncomfortable number:

If underpowered A/B tests are stopped at the first sign of significance, up to 80% of the supposed winning variants are false positives.Martin Goodson: “Most Winning A/B Test Results Are Illusory”, 2014

This explains a phenomenon many website owners know: the tool reports “+15% conversion!” — but nothing arrives in revenue. The win existed only in the sample, not in the world.

Even with enough traffic, the usual winner is: nothing

Suppose you do have the traffic. Then the real game begins — and its odds are documented by the companies running the world's largest experimentation programmes themselves: Ron Kohavi (long-time head of Microsoft's experimentation platform) and Stefan Thomke (Harvard Business School) report in Harvard Business Review (2017) that at Google and Bing only 10 to 20% of all experiments produce positive results. At Microsoft the rule of thirds applies: one third of ideas help, one third do nothing, one third hurt. According to an analysis by former data science executives at O’Reilly (2021), Netflix assumes 90% of its own ideas are wrong; at Booking.com, 9 out of 10 fail.

The consequence: A/B testing only pays off as a permanent programme with dozens of tests per year — which multiplies the traffic requirement again. And even a methodically clean test only answers which variant is better. Why visitors leave — and what you should change next — it never tells you. The hypotheses have to come from somewhere else.

What actually works below 30,000 visitors

The alternative is as old as it is proven: qualitative user testing — watching and listening to real people while they use your website and think aloud.

The crucial point: user tests don't need streams of visitors, they need a handful of suitable people. Jakob Nielsen and Thomas Landauer showed as early as 1993 — and the Nielsen Norman Group still considers it the most economical method today: a single tester uncovers on average about 31% of an interface's usability problems, five testers together about 85%. Each additional tester mostly rediscovers known issues. Hence the recommendation: three small rounds of five testers each (test → fix → test again) rather than one expensive large-scale study. The figure is an average across many projects — very complex websites need more rounds, not more testers per round.

Five testers uncover on average around 85% of a website's usability problems — regardless of whether the site gets 500 or 500,000 visitors a month.Nielsen/Landauer 1993; Nielsen Norman Group 2000

And there is almost always something to find. The Baymard Institute, which has researched e-commerce UX for years, puts the average cart abandonment rate across 49 studies at 70.19% — and estimates that an average large online shop could increase its conversion rate by a good 35% through better checkout design alone. These problems don't show up in any analytics report — but the fifth tester who says out loud “I can't find the shipping costs, that seems fishy” shows them to you in a minute.

This is exactly the path we built Test it Baby for: upload your website or your designs, book for example five testers from the German-speaking panel — or invite your own customers via link — and receive screen recordings with think-aloud commentary plus an AI summary within hours. GDPR-compliant, all data on servers in Germany, billed per answer instead of per visitor count. There is no minimum traffic — it's the method that works from day 1.

When A/B tests are still the right choice

Honesty requires saying: A/B tests are not a bad tool — for most websites they are just the wrong first tool. They make sense when three conditions come together:

• The tested page consistently gets at least 30,000–50,000 visitors per month — otherwise even a simple test takes a quarter.
• There is a clear hypothesis, ideally from user tests (“testers overlook the cart button”) rather than gut feeling.
• There is discipline: calculate the sample size beforehand, fix the duration, don't stop midway.

The strongest combination is the sequence: the user test finds the problem and provides the hypothesis, the A/B test validates the solution — if traffic allows. Doing it the other way round means testing variants of guesswork.

Decision guide: the right tool for your traffic

Frequently asked questions

Sources

1. Jakob Nielsen: Why You Only Need to Test with 5 Users. Nielsen Norman Group, 2000 (based on Nielsen/Landauer, ACM INTERCHI ’93).
2. Ron Kohavi, Stefan Thomke: The Surprising Power of Online Experiments. Harvard Business Review, September–October 2017.
3. Eric Colson, Daragh Sibley, Dave Spiegel: The Sobering Truth About the Impact of Your Business Ideas. O’Reilly Radar, 2021.
4. Martin Goodson: Most Winning A/B Test Results Are Illusory. Whitepaper, 2014.
5. Evan Miller: How Not To Run an A/B Test. 2010.
6. Evan Miller: Sample Size Calculator (two-proportion test, standard tool for A/B sample sizes).
7. Baymard Institute: Cart Abandonment Rate Statistics (continuously updated meta-analysis of 49 studies).
8. Baymard Institute: E-Commerce Checkout Usability — Report & Benchmark.