By Mehdi HassanWhat is a false winner in A/B testing?
A false winner is a variant that looks better in an A/B test but isn't really better. It happens when the test's result reflects noise, a broken setup or a temporary effect rather than a real change in how visitors behave. Teams ship it, conversion doesn't improve, and nobody can explain why.
False winners are worse than no test at all, because they create confidence in the wrong direction. They also pile up: a program that ships several false winners a year ends up with a page that has been "optimized" many times and hasn't really improved. The mistakes below are the ones that cause them most often. For the full process they fit into, see our guide to A/B testing landing pages.
What are the most common AB testing mistakes?
The most common AB testing mistakes fall into three groups: stopping and deciding at the wrong time, setting the test up wrongly, and reading the results wrongly. The table summarizes them; the sections below explain each in plain English.
Mistake | What happens | Fix |
|---|---|---|
Peeking and stopping early | You stop at a random high point and call it a win | Fix the sample in advance and run to it |
Too little traffic | Real effects can't be told apart from noise | Test bigger changes or don't test yet |
Sample ratio mismatch | The split is uneven, so the groups aren't comparable | Stop, find the cause, restart |
Novelty effect | A change wins because it's new, then fades | Run full weeks and watch the trend |
Too many metrics or variants | Something looks significant by chance | One primary metric, few variants |
Build and tracking bugs | You measure flicker or broken events, not the change | QA both versions before launch |
Ignoring downstream quality | More sign-ups, worse customers | Add guardrail metrics |
Why is peeking at an A/B test a problem?
Peeking is a problem because results swing around randomly during a test, and if you check often and stop the first time one version looks significantly ahead, you'll stop on a random high far more often than the significance level suggests. The 5% false positive rate a standard test promises only holds if you look once, at the planned sample.
Think of flipping two fair coins and checking after every flip which has more heads. At some point, one will be well ahead, purely by chance. If you stop at that moment and declare it the "better coin", you've found a false winner. Daily checks on a live test are the same thing with conversion data.
- Decide the sample size before launch, using our guide to A/B test sample size, and don't call the test before it.
- Check early only for breakage: is the split even, are events firing? Not for who's winning.
- If you need to look often, use a tool with sequential testing or Bayesian methods designed for continuous monitoring, and follow its stopping rules exactly.
What is sample ratio mismatch?
Sample ratio mismatch (SRM) is when the number of visitors in each variant is clearly different from the split you set, for example noticeably more visitors in A than in B on a 50/50 test. It means something in the setup is assigning or counting visitors unevenly, so the groups aren't comparable and the result can't be trusted, whatever it says.
Common causes include a variant that loads slower, so more visitors leave before being counted; redirects in split URL tests that lose some visitors; bots treated differently by each version; and tracking that fires on one version but not reliably on the other. Many testing tools flag SRM automatically. If yours doesn't, compare the visitor counts with a chi-square test, or an online SRM calculator, a day or two after launch.
What is the novelty effect in A/B testing?
The novelty effect is when a change performs well at first simply because it's new and draws attention, then fades as visitors get used to it. It mostly affects returning visitors, such as existing users on a pricing page, and it can make a short test show a lift that disappears after launch.
The opposite also happens: a change can underperform at first because returning visitors are used to the old layout, then recover. Both are reasons to run tests in full weeks and to look at the daily trend for each variant at the end of the test. If the lift is shrinking steadily over the test, be cautious. Comparing new and returning visitors separately is a useful check when your tool supports it.
For example, suppose a new banner wins big in week one
For example, suppose a team adds a bright banner announcing a new feature above the pricing table. In the first week, the variant's upgrade rate is well ahead. In week two, the gap narrows, and by week three the two versions are almost level. Most of the early lift came from returning users noticing something new.
If the team had stopped after week one, they'd have shipped a "winner" that does very little. This is a hypothetical example to show the pattern, not a real result.
How do too many metrics and variants create false positives?
Too many metrics and variants create false positives because each comparison has its own chance of a fluke. At a 5% significance level, one comparison in twenty will look significant by chance alone. Check twenty metrics, or run several variants against the control, and the odds that something "wins" by chance alone become high.
- Choose one primary metric before launch, written into the test hypothesis. It alone decides the test.
- Treat secondary metrics as explanation, not as a second chance to find a winner.
- Limit variants. Each extra variant needs its own full sample and adds comparisons.
- If you must compare many, apply a correction for multiple comparisons, which most good tools support.
- Be careful with segments. A variant that "wins on mobile in Germany" after losing overall is usually a fluke unless you planned that segment in advance.
Which build and tracking mistakes ruin tests?
The build and tracking mistakes that ruin tests are flicker, slower variants, broken mobile layouts, conversion events that fire twice or not at all, and internal traffic counted as visitors. They're easy to miss because the testing tool still produces a clean-looking result.
Check for flicker
Load the variant on a slow connection. If the original flashes before the change, visitors are seeing both versions.
Compare speed
Run a performance check on both versions. A heavier variant tests speed, not just your change.
Test every state
Mobile, tablet, desktop, logged in and out, cookie banner accepted and dismissed. Our website QA testing guide has a full checklist.
Verify the conversion event
Complete the conversion on each version and confirm it records once, with the right variant label, in both the testing tool and analytics.
Exclude internal traffic and bots
Filter your team's IPs and known bots, so they don't fill one variant faster than the other.
Qualitative tools help you catch these early. Watching a handful of recordings of each variant shows broken layouts quickly, and our guides to heatmaps and session recordings and Microsoft Clarity vs Hotjar cover the setup. If you're still choosing a testing platform, our comparison of A/B testing tools for startups notes which ones flag SRM and handle flicker.
