By Mehdi HassanDo headline changes make a measurable difference in an A/B test?
Headline changes make a measurable difference when the new headline makes a different promise to the reader. A headline is the first thing visitors use to decide whether the page is for them, so changing what it claims changes who stays. Rewording the same claim rarely moves results enough to detect on a startup's traffic.
This article is a hypothetical walkthrough, not a report of real client results. We don't publish client test data, so instead we show the process we follow on a headline test step by step, with an illustrative seed-stage SaaS company. The numbers inside it are invented to show the method and are labelled as such. For the full testing method, see our guide to A/B testing landing pages.
The one hard data point on copy tests in our research is about buttons, not headlines: an Optimizely test reported that "Watch a Demo" converted at 1.59% against 0.91% for "Get Started". It's a reminder that what the words promise matters more than how they're styled.
What is the setup of this hypothetical headline test?
A hypothetical example: a seed-stage SaaS company sells scheduling software for field service teams. Its Framer landing page gets steady paid and organic traffic, and the main goal is a booked demo. The founder wrote the current headline at launch and it describes the product's features.
Item | Hypothetical value |
|---|---|
Product | Scheduling software for field service teams |
Platform | Framer, with native A/B testing |
Goal | Demo booked (not form started) |
Current headline angle | Mechanism: what the software does |
Constraint | Traffic supports one clean test per quarter |
The constraint shapes everything. With traffic for one test a quarter, the team can't afford a test that ends inconclusive, so the variant has to be a large change. Our A/B test sample size guide explains why: the research we use puts a two-variant test at roughly 15,000 to 20,000 visitors per variant to detect a 15% lift at a 2% baseline.
How do you choose headline angles to test?
You choose headline angles from customer research, not from a brainstorm. The angles worth testing are the different reasons real buyers give for choosing you, because each one attracts a slightly different reader.
Collect buyer language
In the walkthrough, the team reads sales call notes and recent reviews and pulls exact phrases about why customers switched.
Group phrases into angles
Three angles appear: the mechanism (automated scheduling), the outcome (fewer missed appointments), and the pain (no more spreadsheet juggling).
Write one headline per angle
Each headline gets a matching subheadline, because the subheadline does the explaining. Our guide to the landing page subheadline covers that job.
Pick the biggest contrast
The team tests the current mechanism headline against the outcome headline, the angle that differs most from what's live.
Write the hypothesis
Because buyers describe the outcome more often than the features, an outcome headline will increase booked demos. The format comes from our A/B test hypothesis template.
How is a headline AB test set up and run?
A headline AB test is set up as a normal A/B test with one unit changed: headline plus subheadline. Everything else on the page stays identical, so any difference can be traced to the message.
- The variant is built in Framer's native testing, so there's no flicker. Setup is covered in how to A/B test in Framer, Webflow and Next.js.
- The goal is a booked demo tracked in the scheduler, with the variant passed through as a hidden field, not just a click on the button.
- Traffic is split 50/50, and the end date is set on day one from the sample size calculation.
- Both variants are checked on mobile, where long headlines wrap differently and can push the call to action below the fold.
Hypothetical result and how the team reads it
For example, suppose that at the planned end date the outcome headline shows more form starts than the control, and slightly more booked demos, but the demo difference is within the range the team's calculator marks as not yet significant.
The team doesn't call it a win on form starts, because form starts weren't the goal. It also doesn't call it a loss. It records the outcome angle as promising, keeps it live because it performed at least as well and matches how buyers talk, and plans the next test around the pain angle.
These figures are illustrative. The point is the reading: judge the agreed goal, respect the planned end date, and treat inconclusive as information.
What can go wrong when you test headlines?
Most headline tests fail because of the setup, not the copy. The common failures are stopping early, judging on the wrong metric and changing other things mid-test.
- Stopping the day a winner appears. Early leads often shrink. Our article on A/B testing mistakes that produce false winners explains why.
- Optimizing for clicks. A vaguer, more curious headline can raise clicks and lower qualified demos.
- Moving traffic sources mid-test. A new ad campaign changes who arrives and makes the comparison unfair.
- Testing headlines that don't match the ad. If the ad promises one thing and the variant another, you're testing message match, not the headline.
What do headline tests teach beyond the winner?
Headline tests teach you which promise your buyers respond to, and that lesson travels beyond the landing page. The winning angle can become the lead message in ads, sales emails and the homepage.
A losing angle is useful too: it tells you which reason to buy isn't strong enough to lead with, even if customers mention it. Over a few quarters, the team in our walkthrough builds a ranked list of angles grounded in evidence instead of opinion. If you want a starting set of angles to write from, our landing page headline formulas and step-by-step landing page copy process are where we'd begin.
Should you test your headline at all?
You should test your headline if your page gets enough traffic to finish a bold test within a quarter and you have a genuinely different angle to try. If you don't have either, rewrite the headline from customer research, ship it, and watch the trend.
