What is an A/B test?

A/B test: definition and purpose

In an A/B test, version A (the current one) is tested against version B, which contains exactly one deliberate change. Visitors are assigned at random so that both groups are comparable. In the end, real user data shows which version generates more inquiries, purchases or clicks—instead of opinions or gut feeling deciding.

Typical test subjects are headlines, images, button text, form length, price displays or newsletter subject lines. A related method is the multivariate test, which combines several elements at once but needs far more visitors. For most mid-sized company websites, the classic A/B test with a single change is the more practical choice.

How an A/B test works

It starts with a hypothesis, for example: “A button saying ‘Request a free quote’ leads to more inquiries than ‘Submit’.” Next, the target metric is defined, such as the conversion rate. Testing software splits the traffic at random, counts the results for each version and calculates whether the difference is reliable or could be down to chance.

Duration is critical. A test should run for at least full weeks to even out weekday effects, and it should only end once the visitor number calculated in advance has been reached. Anyone who checks in between and stops at the first lead often makes wrong decisions, because early swings frequently disappear again.

Real-world example: a tradesperson’s inquiry form

A tiling contractor suspects that its contact form with eight required fields puts prospects off. Version B asks only for name, phone number and the service wanted; details are clarified in a callback. Both versions run in parallel for several weeks, and the metric is submitted inquiries per hundred visitors. Everything else on the page stays unchanged, so the difference can clearly be attributed to the form.

If the short version shows a clear lead, it becomes the new standard and the next test follows, for example on the headline. If there is no difference, that is a result too: the form was apparently not the problem, and the effort goes into other levers such as load time or trust elements.

Limits, common mistakes and imple­menta­tion

Meaningful tests need enough visitors. With a conversion rate of around two percent, small differences often require several thousand to a few tens of thousands of visits per version. Sites with little traffic are better off testing bigger changes or relying on user feedback first. Measurement and consent also need to be sorted out: depending on the tool, consent may be required.

Nano Concepts sets up A/B tests cleanly, makes sure the testing tool does not slow your site down, aligns measurement and cookie consent, and evaluates the results together with you, so that insights turn into concrete improvements to your site and processes.

• Only one change per test, otherwise the cause stays unclear
• Define the hypothesis and target metric before starting
• Plan duration and visitor numbers in advance; don’t stop early
• Don’t run several tests on the same page at the same time
• Document the results—including tests without a winner