By Mehdi HassanWhat is A/B testing on a landing page?
A/B testing on a landing page is an experiment where visitors are randomly split between the current page (the control, A) and a changed version (the variant, B), and you compare how many in each group complete the goal, such as a sign-up or a demo request. Because the split is random and simultaneous, the difference in results can be attributed to the change rather than to seasonality or a different traffic mix.
That's what makes it different from "we changed the page and conversions went up". A before-and-after comparison mixes your change with everything else that happened that month. A properly run A/B test isolates it. The catch is that "properly run" hides a lot of work, and most guides cover only the statistics. This guide covers the whole process, including the build side most teams underestimate, and links to a deeper article for each step.
How does AB testing on landing pages work, step by step?
AB testing on landing pages works as a fixed sequence: research, hypothesis, sample size, build, QA, launch, wait, analyze, decide. Skipping any step is how teams end up shipping "winners" that were never real.
Research the page
Look at analytics, heatmaps, session recordings, sales call notes and support questions to find where and why visitors drop off.
Write a hypothesis
State what you'll change, what you expect to happen and why, tied to what you observed. Our A/B test hypothesis template forces that link.
Check traffic and set the sample size
Calculate how many visitors per variant you need for the smallest lift you care about, and how long that takes at current traffic.
Build the variant
Make the change in your testing tool, CMS or code, keeping everything else identical.
QA both versions
Check every device, browser and state, that tracking fires once per conversion, and that the split is roughly even.
Run to the planned sample
Launch, then leave it alone. Run full weeks, so weekday and weekend behavior are both included.
Analyze and decide
Compare the primary metric at the planned sample, check secondary metrics for harm, then ship, iterate or discard.
Write it down
Record the hypothesis, result and what you learned, including losers. The archive is worth more than any single test.
The rest of this guide goes through the parts of that sequence that cause the most trouble: traffic, what to test, building variants, tracking, reading results and choosing a tool.
Do you have enough traffic to A/B test a landing page?
Many startup landing pages don't have enough traffic for a valid A/B test, and it's the first thing to check. The required sample depends on your current conversion rate and the smallest improvement you want to detect, and at typical startup conversion rates it's measured in tens of thousands of visitors per variant.
- Visitors per variant to detect a 15% lift at a 2% baseline, per Spotler's guide to A/B test traffic
- 15,000–20,000
- Median landing page conversion rate across industries in Unbounce's Conversion Benchmark Report
- 6.6%
- Unbounce's median for SaaS landing pages in the same report
- About 3.8%
Lower baseline rates and smaller lifts need much bigger samples, and the exact figure also depends on your confidence and power settings. Our guide to A/B test sample size walks through worked examples at startup traffic levels and gives a clear "don't test yet" threshold.
What should you A/B test first on a landing page?
Test the things that change what visitors understand or are offered first: the headline and value proposition, the offer itself (trial, demo, free plan), the page structure and the proof. Those changes can move conversion enough to detect with realistic traffic. Small cosmetic tweaks rarely do.
Test area | Typical potential impact | Traffic needed to detect it | Example |
|---|---|---|---|
Offer and CTA path | High | Lower (big effects are easier to see) | Free trial vs "Watch a demo" as the primary action |
Headline and value proposition | High | Lower to medium | Outcome-led headline vs category-led headline |
Page structure and length | Medium to high | Medium | Short page vs long page with objection handling |
Proof and its placement | Medium | Medium | Customer quote next to the form vs logo strip |
Form length and fields | Medium | Medium | Four fields vs two-step form |
Visual details | Low | High (small effects need big samples) | Button color, icon style, image swap |
One data point shows why the offer and the label matter: in an Optimizely test reported by Convertica, a "Watch a Demo" CTA converted at 1.59% against 0.91% for "Get Started". That's one test on one site, not a rule, but it's the kind of change worth testing before anything cosmetic. Our list of 40 A/B test ideas for SaaS landing pages ranks ideas by likely impact and traffic needed, and our breakdown of headline tests on SaaS landing pages goes deep on the highest-impact area.
How do you build A/B test variants without breaking the test?
Build variants so that the only difference a visitor experiences is the change you're testing. That sounds obvious, but flicker, slower load times, layout shifts and broken mobile states are all differences too, and they can decide the result on their own.
- Avoid flicker. Client-side tools that swap content after the page loads can flash the original version first. Use the tool's anti-flicker setup, or render variants on the server or at the edge.
- Match performance. If the variant loads a heavier image or an extra script, you're testing speed as well as the change.
- Build it like production. A variant hacked together in a visual editor often breaks on smaller screens or in states like logged-in visitors and cookie banners.
- Keep assignment sticky. A returning visitor must see the same version every time, or your data mixes groups.
- Exclude internal traffic and bots, so your own team's visits don't count.
How you build depends on the platform. Visual builders, CMS-based variants and code-level flags each have trade-offs, which we compare in our guide on how to A/B test in Framer, Webflow and Next.js. If your marketing site runs on Next.js, server-side assignment with edge middleware avoids flicker entirely; our guide to Next.js A/B testing with edge middleware and flags shows the setup.
A QA checklist for every variant before launch
Before launch, check both versions on desktop and mobile in at least two browsers, with and without the cookie banner accepted. Submit the form or complete the conversion on each version and confirm the event fires exactly once, with the right variant label. Check the page speed of both versions. Confirm the split in your tool's preview or debug mode, and that internal IPs are excluded.
After launch, look at the first day's data only to confirm the split is roughly even and conversions are being recorded. Don't look at which version is winning. If the split is clearly uneven, stop and fix it, because that's a sign the test is broken, not that one version is better.
How should you track conversions in an A/B test?
Track one primary conversion that the variant is meant to change, plus a few secondary and guardrail metrics. The primary metric decides the test; the others tell you whether the winner causes harm somewhere else, such as more sign-ups but fewer activated users.
- Primary metric: the action the hypothesis predicts will change, such as demo requests or trial sign-ups, counted once per visitor.
- Secondary metrics: steps before it (clicks on the CTA, form starts) to explain why the primary moved.
- Guardrail metrics: downstream quality such as activation, qualified pipeline or refunds, so you don't optimize for junk leads.
- Variant labels on every event, sent to your analytics tool as well as the testing tool, so you can check one against the other.
Set this up before the test, not during it. Our guide to landing page conversion tracking with GA4 events covers the event setup, and our explainer on bounce rate vs engagement rate explains why neither of those should be your primary test metric.
How do you read A/B test results?
Read A/B test results at the sample size you planned, on the primary metric you chose, and with the uncertainty in view. A result is worth acting on when the difference is statistically significant at your chosen threshold, the confidence interval doesn't include a meaningful loss, and the guardrail metrics haven't moved the wrong way.
The most common way to get a false winner is peeking: checking results every day and stopping the moment one version looks ahead. Random noise makes one version look ahead at some point in almost every test, so stopping on that moment inflates your false positive rate. Novelty effects, uneven splits (sample ratio mismatch) and running many variants or metrics at once cause similar problems. We explain each one in plain English in A/B testing mistakes that produce false winners.
A/B testing vs multivariate vs split URL testing: which should you use?
Use a standard A/B test for almost everything on a startup landing page, a split URL test when the variant is a different page or a full redesign, and multivariate testing only when you have very high traffic and want to see how several elements interact.
Method | What changes | Traffic needed | Best for |
|---|---|---|---|
A/B test | One version vs another on the same URL | Moderate to high | Headlines, offers, sections, forms |
Split URL test | Traffic split between two different URLs | Moderate to high | Full redesigns, new templates, different builds |
Multivariate test | Several elements at once, in combinations | Very high | Mature, high-traffic pages fine-tuning details |
A/B/n test | Three or more versions at once | Higher than A/B (each variant needs its own sample) | Comparing a few distinct headline or offer options |
The full comparison, with examples of when each one misleads, is in A/B testing vs multivariate testing vs split URL testing. Personalization is a related but different idea: instead of finding one winner for everyone, you show different versions to different segments. Our guide to landing page personalization covers when that's worth the effort.
Which A/B testing tools should a SaaS team use?
Choose an A/B testing tool by how your site is built, how much traffic you have and who will run the tests. Marketer-friendly visual tools suit teams without developers on hand, while product analytics platforms with feature flags suit teams that already use them and want tests close to the code.
- Visual testing platforms (VWO, Optimizely and similar) let marketers build variants without code, at the cost of client-side flicker risk and higher plan prices.
- Product analytics with experiments (PostHog and similar) combine flags, experiments and analytics, which is efficient if the team is comfortable with a bit of code.
- Platform-native options in some website builders handle simple tests inside the CMS. Check each platform's current plan page for what's included.
- Custom server-side tests on frameworks like Next.js give full control and no flicker, but need engineering time and your own analysis.
Plans and features change often, so compare current pricing pages before committing. Our roundup of the best A/B testing tools for startups compares VWO, Optimizely, PostHog and others on price, setup and stats approach.
Where should you go next?
Go next to the article that matches the step you're stuck on. Each one goes deeper than this overview, with worked examples and templates.
If you want to... | Read |
|---|---|
Check whether you have enough traffic | |
Find what to test first | |
Write a hypothesis tied to research | |
Avoid false winners | |
Pick a tool | |
Run tests on your platform | |
Choose a testing method | |
See what headline tests teach | |
Get inspiration for variants |
For page-type specifics, our guides to B2B services landing pages and developer tool landing page examples show which sections tend to carry the most weight for those audiences, which is a good source of test ideas.
