By Mehdi HassanWhat is the difference between A/B, multivariate and split URL testing?
An A/B test shows visitors one of two versions of a page that differ in one idea. A multivariate test changes several elements at once and measures every combination. A split URL test sends visitors to two different URLs, usually two entirely different pages. All three are controlled experiments; they differ in what they vary and how much traffic they need.
Each method sits inside a wider testing program, which our guide to A/B testing landing pages lays out from hypothesis to decision. Here we compare the three methods side by side so you can pick one for the test in front of you.
A/B test | Multivariate test | Split URL test | |
|---|---|---|---|
What changes | One idea, two versions | Several elements, all combinations | Two whole pages at different URLs |
Question answered | Is B better than A? | Which elements matter, and do they interact? | Is page B better than page A? |
Traffic needed | Baseline | Baseline times the number of combinations | Baseline |
Best for | Headlines, offers, hero sections | High-traffic pages with many small elements | Redesigns, new templates, platform changes |
Setup effort | Low | Medium to high | Medium |
What you don't learn | Why B won | Little, if traffic is enough | Which part of the new page made the difference |
How much traffic does each type of test need?
An A/B test needs enough visitors for each of its two versions, a multivariate test needs that amount for every combination, and a split URL test needs the same as an A/B test. Traffic, not preference, usually decides the method.
The research behind our landing page work puts the bar for a two-variant test at roughly 15,000 to 20,000 visitors per variant to detect a 15% lift from a 2% baseline conversion rate. That's the baseline. A multivariate test with two headlines and two hero images has four combinations, so it needs roughly that figure for each of the four. Add a third element with two options and you have eight.
- Visitors per variant to detect a 15% lift at a 2% baseline
- 15–20k
- Groups to fill in a simple A/B or split URL test
- 2
- Groups to fill in a 2 x 2 x 2 multivariate test
- 8
Smaller lifts need more visitors; bigger lifts need fewer. If your page gets a few thousand visitors a month, a bold A/B or split URL test is the only kind that will finish this year. Our explainer on A/B test sample size shows how to calculate the number for your own baseline.
When should you use an A/B test?
Use an A/B test when you have one clear idea to check, such as a new headline angle, a different offer or a reworked hero. It's the default method for most landing pages because it needs the least traffic and gives the cleanest answer.
The weakness is that an A/B test tells you which version won, not why. Pair it with a written hypothesis so the result teaches you something about your buyers. Our A/B test hypothesis template gives you the format, and our list of A/B test ideas ranked by likely impact helps you choose which idea is worth the traffic.
When is a multivariate test worth it?
A multivariate test is worth it when a page has very high traffic and you need to know which of several elements drives results, and whether they interact. A new headline might only work with a matching image, and only a multivariate test shows that.
In practice that means checkout pages at large e-commerce sites, sign-up flows at consumer apps, or ad landing pages with heavy paid traffic. For a typical B2B SaaS landing page, a multivariate test spreads visitors so thin that none of the combinations reaches a result. Teams then read noise as insight, which is one of the A/B testing mistakes that produce false winners.
Pros
- Shows which elements matter most
- Reveals interactions between elements
- Tests many ideas in one experiment
Cons
- Needs traffic multiplied by every combination
- Takes much longer to finish
- Easy to misread when sample sizes are small
- Hard to QA every combination properly
When should you use a split URL test?
Use a split URL test, sometimes called a redirect test, when the two versions are too different to build as edits of one page. That covers redesigns, new page templates, a new layout from a different builder, or a long page against a short one.
Split URL tests suit low-traffic startups well, because a whole new page is a big change and big changes are easier to detect. The trade-off is that you learn which page won, not which part. Before running one, read our setup notes in how to A/B test in Framer, Webflow and Next.js, and check both pages with the same care you'd give a launch, as in our website QA testing checklist.
Hypothetical example: choosing a method for three different pages
For example, suppose a startup has three pages it wants to improve.
The pricing page gets about 3,000 visitors a month. A multivariate test is out of reach, and even a small A/B test would take many months. The team runs a split URL test of a simplified pricing layout against the current one, accepting that it will take most of a quarter.
The main paid-ads landing page gets about 40,000 visitors a month. That's enough for a two-variant A/B test of a new headline angle within a few weeks, but not for a multivariate test of headline, image and button together.
The homepage is being redesigned. Rather than testing the redesign piece by piece, the team runs a split URL test of old against new, then follows up with A/B tests on the new page.
What can you do if you don't have enough traffic for any test?
If you don't have enough traffic for any test, improve the page through research instead: heatmaps, session recordings, short on-site surveys and customer interviews. These methods show why visitors hesitate without needing statistical significance.
Start with heatmaps and session recordings to see where attention drops, then run a simple comprehension check like the 5-second test on your hero. Ship the improved page, watch the trend over a few weeks, and save formal testing for when traffic grows.
