Setting up a landing page comparison the right way

Why “which one do you like better?” is the wrong question

The most common mistake in design comparisons is the taste question. It produces answers — just not useful ones. The reason is well researched: under the aesthetic-usability effect (Nielsen Norman Group), people perceive beautiful interfaces as more usable than they actually are — the prettier design wins the taste question even when visitors cannot find the price on it. On top of that: the aesthetic first impression forms within about 50 milliseconds (Lindgaard et al., 2006) — long before anyone has read a single line.

A useful comparison therefore measures not liking but three things: comprehension (do visitors grasp within seconds what is offered?), findability (do they reach price, contact, CTA without searching?) and trust (would they enter data or buy here?). The latter is no side issue: 46.1% of people judge a website's credibility by its visual appearance (Stanford Web Credibility Project).

The comparison test in six steps

1. Fix the decision questionOne question, one sentence: “Which draft gets first-time visitors to an enquiry faster?” Everything in the test serves this question — otherwise you will be back to debating colours afterwards.

2. Make both drafts testableLive URL, staging or uploaded design drafts — what matters is that testers can genuinely view and use both variants. One advantage over A/B testing: none of it has to be public.

3. Capture the first impression first (5-second test)Show a variant, hide it after five seconds, ask: “What is offered here, for whom?” Only then allow free exploration. This separates the 50 ms impression from the rest.

4. Identical tasks on both variantsTwo to three realistic tasks, worded identically for A and B: “Find out what the offer costs.” “Request a quote.” Testers think aloud — every hesitation is a data point.

5. Rotate the orderHalf of the testers start with A, half with B. Otherwise the first draft seen shapes the judgement of both.

6. At the end: the comparison question with reasoningOnly now the direct choice: “On which page would you be more likely to enquire — and why?” The reasoning matters more than the choice itself.

The aesthetic first impression of a website forms in around 50 milliseconds — so both must be tested: the instant impression and behaviour on real tasks.Lindgaard et al., Behaviour & Information Technology, 2006

Evaluating: count, quote, decide

Per variant and task, note three things: How many testers completed the task? Where exactly did they hesitate or take a wrong turn? And which verbatim quotes explain the behaviour? After around 5 testers, the picture is almost always clear in practice — not as a percentage, but as a pattern: draft B is understood faster, but its contact path gets overlooked. The result is rarely “A wins outright”; it is usually: B's structure with A's pricing display. That combination is the real prize of the qualitative comparison — an A/B test would have given you only a number.

And if you later want to know how much the winning variant delivers quantitatively: that is the moment for a classic A/B test — provided your traffic supplies the required 30,000 visitors. In this order it is also far more likely to succeed: you are then testing an evidenced hypothesis instead of a guess.

How the comparison works with Test it Baby

For exactly this scenario we offer a ready-made format: the guided A/B comparison. You upload both drafts (finished pages or design files — we host them for the test), define the tasks once, and the platform guides each tester through both variants in sequence — including a final comparison with a reasoning field. Testers come from the DACH panel or from your own customer base; recordings, transcripts and the AI summary per variant are usually ready the same day. GDPR-compliant, servers in Germany.

Frequently asked questions

How do I compare two landing page drafts without an A/B test?

With a qualitative comparison test: around 5 testers see both drafts (in rotating order), complete the same tasks on each and think aloud. You evaluate not “which one do you like better” but where testers understand the offer faster and where they hesitate. This works without live traffic — even with drafts that are not online yet.

Why shouldn't I just ask which draft people prefer?

Because liking and working are two different things: the aesthetic-usability effect (Nielsen Norman Group) shows that beautiful designs are perceived as more usable than they are — taste judgements mask real problems. The comparison only becomes meaningful through tasks (“find the price”) and comprehension questions (“what is offered here?”).

In which order should testers see the variants?

Alternating: half of the testers start with draft A, the other half with B. The first draft seen shapes expectations and would otherwise systematically perform differently — rotating balances out this order effect.

Is such a comparison enough to base a decision on?

For the design decision between two drafts: yes — it shows which draft is understood faster and has fewer stumbling blocks, and above all why. What it does not deliver is a conversion forecast in percent; that would later require an A/B test with tens of thousands of visitors. The best sequence: decide and improve qualitatively, validate quantitatively when traffic allows.

Sources

Nielsen Norman Group: The Aesthetic-Usability Effect.
Gitte Lindgaard et al.: Attention Web Designers: You Have 50 Milliseconds to Make a Good First Impression! Behaviour & Information Technology, 2006.
B. J. Fogg et al.: How Do Users Evaluate the Credibility of Web Sites? Stanford Web Credibility Project, 2003.
Jakob Nielsen: Why You Only Need to Test with 5 Users. Nielsen Norman Group, 2000.