Conversion testing

How to validate improvements with conversion testing before implementing them permanently

Conversion testing is the method of validating that changes improve the conversion rate before they are implemented permanently. Instead of guessing or relying on best practice, changes are tested systematically with real users.

The most common format is the A/B test: some visitors see the original version (A), others see the new version (B). Once there is enough data, the results are compared. If the difference is larger than chance can reasonably explain, the better version wins and becomes permanent.

Without testing, conversion rate optimisation is risky. Changes that seem good can in fact reduce conversion. Testing reduces the guesswork, and decisions are based on real results from users.

What is conversion testing?

Definition

Conversion testing (A/B testing) is the process of comparing two or more versions of a web page to determine which one gives the highest conversion rate.

Statistical significance: When the test is complete, you calculate whether the difference is larger than chance can reasonably explain.

Why test instead of simply implementing?

The problem without testing

Scenario

The analysis shows that a CTA button is not clearly visible. Decision: change the colour from grey to red.

Implementing directly

The colour is changed across the whole site. But nobody knows whether red works better. Perhaps it makes things worse?

With an A/B test

50 per cent see the grey button (A), 50 per cent see the red button (B). After two weeks, the data shows that red converts 15 per cent better. Now there is evidence to implement the red button permanently.

Alternative scenario

The test shows that red in fact converts 10 per cent worse. Without the test, a deterioration would have been implemented permanently.

Result

Testing reduces the risk of rolling out a change for the worse.

When should you test?

Which types of A/B test can you run?

A/B test

Compares two versions: the original (A) vs a variant (B).

One change at a time

To know what affected the result, only one thing is changed.

Result

If B wins, we know that the red button works better. If several things have been changed at once, it is impossible to know what caused the change.

A/B/n test (several variants of the same element)

Compares the original with several variants.

Use

Test several alternatives at once to find the best one faster.

Requirements

More traffic is needed (the traffic is split across more variants). Multiple comparisons also require a correction in the analysis, otherwise false winners become more common.

Multivariate testing (MVT)

Tests several elements at once in different combinations.

Example

Combinations

2 × 2 × 2 = 8 variants tested at the same time.

Use

Understand how elements interact. Perhaps a red button works better with headline A but a green button works better with headline B.

Requirements

Very high traffic (the traffic is split across many variants). Often only for large sites.

Split URL test

Instead of changing elements on the same URL, completely different pages are tested.

How to run an A/B test, from hypothesis to result

Step 1: Formulate a hypothesis

A hypothesis is a prediction of what will happen and why.

Format

“If we [change], [audience] will [result] because [reason].”

Examples

Hypothesis 1

“If we add customer reviews to product pages, more visitors will add products to the basket because reviews build trust and reduce uncertainty.”

Hypothesis 2

“If we shorten the checkout form from 12 fields to 6 fields, more people will complete their purchase because shorter forms reduce friction.”

Hypothesis 3

“If we change the CTA from ‘Read more’ to ‘Get a free demo’, more people will click because ‘free demo’ communicates the value more clearly.”

Why a hypothesis matters

Without a hypothesis

Tests become random. “Let’s test a red button because it might work?”

With a hypothesis

Tests are targeted and based on analysis. “The analysis shows that people don’t see the button. A red colour stands out more from the background, so more people should see it and click on it.”

Step 2: Define the test

What is tested

Where it is tested

Which page? (Product page, landing page, checkout?)

Primary goal

What is measured? (Add-to-cart rate, checkout completion, clicks on the CTA?)

Secondary goals

Other metrics to monitor (bounce rate, time on page).

Example test

Test

Add customer reviews to product pages

Hypothesis

Reviews build trust and increase the add-to-cart rate.

Version A

Product page without reviews (current)

Version B

Product page with a Trustpilot widget (average rating and the five most recent reviews)

Page

All product pages

Primary goal

Add-to-cart rate

Secondary goals

Time on product page, bounce rate

Step 3: Create variants

Design and development

Design the variant

A mock-up or wireframe of the change.

Develop the variant

Build the alternative version in your own codebase or content management system, in the same way as any other change on the site.

QA test

Make sure the variant works technically (no broken links, responsive, fast).

Step 4: Set up the test

The test is built into your existing systems.

For each visitor, the variant shown is stored together with your own data, which is then extracted for a separate analysis.

Configuration

Traffic allocation

How much traffic goes to each variant? Typically 50/50.

Audience

All visitors or specific segments? (New vs returning, desktop vs mobile)

Test duration

How long should the test run? Calculate the duration before you start, based on the traffic, the current conversion rate and the smallest effect you want to be able to detect.

Tracking

Make sure conversions are tracked correctly for both variants.

Step 5: Run the test

Wait until there is enough data

Avoid stopping too early

Mistake

Seeing that B is ahead after two days and ending the test.

Problem

Not enough data. The difference may be chance.

Right

Let the test run for the duration decided before you started.

Avoid “peeking” too often

Mistake

Checking the results every day and getting worried if B performs worse on a given day.

Problem

It creates a risk of stopping the test too early, and every reading that could stop the test makes false winners more common.

Right

Check that the measurement is working, but do not read off the result or stop the test before the set time has passed.

Step 6: Analyse the results

Statistical significance

What it means

A measure of whether the difference between A and B is larger than chance can reasonably explain.

Confidence level

Usually 95 per cent (p-value below 0.05).

Interpretation

If there is in reality no difference, a difference as large or larger would occur in at most 5 out of 100 such tests. The p-value does not tell you how likely it is that B is better.

Example

Version A

1,000 visitors, 20 conversions (2.0 per cent)
Version B: 1,000 visitors, 30 conversions (3.0 per cent)

Difference

+1 percentage point (50 per cent relative improvement)

Statistical significance

Not reached (p = 0.15 in a two-sided two-sample test of proportions)

Interpretation

The difference may be chance, and 20 and 30 conversions respectively are too little evidence. Do not roll out B on the basis of this result.

Calculate the significance

Extract the data for each variant and do the calculation in a separate step. The test duration and sample size are decided in advance.

Evaluate secondary goals

Scenario

B increases the add-to-cart rate by 20 per cent but also increases the bounce rate by 30 per cent.

Analysis

Perhaps B attracts the wrong audience? Or creates confusion?

Decision

Investigate further before implementing.

Step 7: Implement or iterate

Common test scenarios

Testing the CTA (call to action)

Example hypothesis: “The button is not noticeable enough. A larger size and a contrasting colour will increase the number of clicks.”

Testing the headline/value proposition

Example hypothesis: “The current headline focuses on features. A headline about benefits evokes stronger feelings and increases conversion.”

Testing images

Example hypothesis: “A lifestyle image helps users picture themselves with the product and increases the desire to buy.”

Testing trust signals

Example hypothesis: “Reviews build trust among new visitors and reduce hesitation before the purchase.”

Testing forms

Example hypothesis: “A shorter form (six fields instead of twelve) reduces friction and gets more people to fill in the whole form.”

Testing pricing

Example hypothesis: “A monthly price looks cheaper than an annual price and gets more people to sign up.”

Testing the checkout flow

Example hypothesis: “Buying without an account reduces friction for new customers and gets more people to complete their purchase.”

Common testing mistakes

Mistake 1: Testing too many things at once

Problem

Changing the headline and the CTA colour and adding reviews in the same test.

Result

If B wins, what caused the change? The headline? The colour? The reviews? Impossible to know.

Solution

Test one thing at a time (or use MVT if there is enough traffic).

Mistake 2: Stopping too early

Problem

Ending the test after two days because B is ahead by 5 per cent.

Result

Not statistically significant. The difference may be chance.

Solution

Let the test run until the duration and sample size decided before you started have been reached.

Mistake 3: Ignoring seasonal variation

Problem

Running a test over a weekend or during a campaign period (Black Friday).

Result

The data is skewed by atypical behaviour.

Solution

Run tests for at least one full week (capture both weekdays and weekends). Avoid starting tests during campaigns.

Mistake 4: The wrong audience

Problem

Testing a new checkout flow, but only for desktop users.

Result

No insight into mobile (where the majority of traffic may be).

Solution

Include all relevant traffic or run separate tests per segment (desktop vs mobile).

Mistake 5: No follow-up

Problem

B wins, is implemented and is then forgotten.

Result

Nobody learns what worked or applies the insights to other parts of the site.

Solution

Document every test. Share insights with the team. Build a knowledge base.

A workflow for continuous testing

Step 1: Create a testing plan

Create a test backlog: An ordered list of tests to run.

Example

Step 2: Define each test

For each test in the plan

Hypothesis

What is expected to happen, and why?

Variant

What exactly is changed?

Metric

What is measured (primary and secondary)?

Success criterion

What counts as success? (for example at least a 10 per cent improvement at a 95 per cent confidence level)

Step 3: Design proposals

Design the variants

Create several alternatives: If there is uncertainty, create two or three variants to test (A/B/C test).

Step 4: Setting up the test

Technical implementation

The test is built in your own codebase or content management system.

QA: Check that both versions work technically.

Step 5: Follow-up

Step 6: Conclusions for the next test

Example

Test 1

Add reviews → +15 per cent add-to-cart (success)

Insight

Trust matters for conversion.

Next test

Test adding guarantees as well (“30-day money-back guarantee”) → Hypothesis: Further trust-building increases conversion more.

A/B tests are built into your own systems

The code and content on a web page often come from several systems. An external testing tool usually only changes the finished page in the browser and does not reach the systems behind it, so it is rarely enough for testing changes made in those systems.

Instead, we build the test into the system you already have, so that the variants are published in the same way as the rest of the site.

The data is then extracted and analysed separately, so that the result is based on your own figures.

It requires development resources, because the test becomes part of the site.

From testing to continuous improvement

Conversion testing is an ongoing process rather than a one-off project.

Cycle

1. Analyse → Identify problems
2. Hypothesis → Formulate what could improve things
3. Test → Validate the hypothesis
4. Implement → Roll out the winner
5. Learn → Document insights
6. Iterate → The next test

Repeat continuously.

Contact us about conversion testing

Carnaby helps companies set up and run systematic A/B tests, from formulating hypotheses to statistical analysis and implementing the winning variants.

Contact us →