A/B testing: understand, test, optimise (without playing the casino)

In digital marketing, making decisions on gut feeling is expensive: you roll out a new version, keep your fingers crossed, then discover the conversion rate has dropped… Oops. A/B testing (also called split testing) is a marketing and experimentation technique that lets you compare two versions of the same element (an original vs a variant, or variant B) to find out which one is more effective. In plain words: you test in a concrete way, you measure a result and you optimise.

What exactly is A/B testing?

A/B testing means randomly showing a visitor (or user) either an original page or a modified page (a variant), then comparing their performance against a specific goal (for example conversion rate, click-through rate, subscription rate, bounce rate, time spent, sign-ups, purchases, etc.).

In one sentence: testing is a technique where you offer a version A and a version B, then work out which one performs better (or, if you want to impress at a dinner party, “see which one performs better”).

A/B, A/B/n, split URL, multivariate: what is the difference?

• A/B test: 2 versions of the same content.
• A/B/n: more than 2 versions (A/B/C…).
• Split testing (split URL): two different URLs (useful when the page changes a lot).
• Multivariate testing: you test a combination of several variables (for example title + image + button). Powerful, but it needs a large number of visits (watch your sample size).

Why do we use A/B testing?

On a web project (or an app), there are often a thousand optimisation ideas: change a piece of content, tweak a button, try a new image, rework a sign-up step, adjust a product presentation… The problem is that between “it looks better to me” and “it is actually more effective” there is a gulf.

A/B testing exists precisely to bridge that gulf: it is a process that lets you determine, based on data, which version works best for a given audience.

The principle is simple: you take a page (or an element), create a version A (the reference version) and a version B (modified with one precise change), then split the traffic randomly between the two. Then you analyse the result through a clear indicator: conversion rate, click rate, sign-up rate, time spent, etc.

To avoid getting lost, each variant is often given an identifier (for example “CTA_red_V1”, “HeroImage_V2”): it helps the team keep track, trace changes and document decisions, especially when several tests run “in parallel”.

The (really useful) benefits of A/B testing

1) Optimise the conversion rate… with hard facts
The most direct benefit is optimising the conversion rate (sign-up, purchase, contact request). Instead of redesigning a site on gut feeling, you move forward with concrete proof. Example: on a campaign landing page, you can compare two wordings of the value proposition, or two visuals. Sometimes a micro change (a clearer title, a reassuring block, a more relevant image) is enough to improve the rate, without changing the overall look.

And above all, you can answer a real business question. Not “is it prettier?” but “does it convert more?”.

2) Reduce risk before rolling out a new version
Rolling out a new version to production is always a risk: you can lower conversions, break a journey or create invisible friction. A/B testing lets you validate a new approach before generalising it. In practice, you test on a group of users, observe, and only then decide.

It is particularly useful when you touch sensitive pages (basket, checkout, forms) or when the team is in the middle of developing a feature. Rather than pushing everything out “in one go”, you play it safe: test → analysis → decision → rollout.

3) Better understand audience behaviour and preferences
A test is not only there to “win”. It is also there to learn. By comparing a version A and a version B, you reveal differences in preference depending on the audience: mobile vs desktop, new visitors vs regulars, SEO traffic vs paid search…

The result: you better understand why people click, drop off or complete. That guides the next steps of the optimisation process: you stop guessing and build a stronger marketing strategy.

4) Improve the user experience and the customer experience
A good A/B test is not only about “more sales”. It can aim at a better experience: less confusion, less effort, more clarity. For example: simplifying a form, prioritising content better, making a page more readable, or making important information more visible.

And behind the user experience, there is the customer experience: a smoother journey, simpler choices, a feeling of trust. These are qualitative gains that often translate into quantitative gains (a better rate, fewer drop-offs, more satisfaction).

5) Improve return on investment: invest where it works
Without testing, you spend time (and sometimes budget) on changes that bring nothing. With A/B testing, you can use your resources more wisely: you test, you measure, then you decide what goes into the final version.

This is true on a website, but also on other channels: an email (subject line, CTA, structure), a campaign page, a product page… In the end, the aim is simple: get measurable results and repeat what works, rather than multiplying redesigns at random.

The role of tools and the “testing solution”

In practice, you often rely on an A/B testing tool (a testing solution) that handles:

• random traffic allocation,
• conversion attribution,
• real-time tracking of results,
• management of several tests across different websites,
• and sometimes the testing and personalisation duo (adapting content to segments).

On the implementation side, there is often a little code involved (depending on the solution and the level of customisation), but the idea remains the same: make the test reliable and usable.

When A/B testing is useful (and when it is not)

Perfect cases

• Landing page / destination page (paid search, campaigns).
• Key pages of a website: product page, basket, sign-up form.
• Emailing: subject line, CTA, layout.
• Mobile app: onboarding, paywall, key screens.

Cases to avoid (or to frame tightly)

• Traffic too low: impossible to get statistically significant results.
• Unstable tracking: if Google Analytics is not reliable, your test is not either.
• Multiple changes at the same time: you no longer know what moved the KPI.
• Pages “under construction”: if the dev team changes the page every day, the test becomes a puzzle.

A/B testing fundamentals: what you test (and what you must not mix)

In digital marketing, A/B testing (sometimes called split testing) has become an essential method for optimising a conversion rate without relying on guesswork. In plain words: testing is a decision-making technique based on data rather than intuition.

It is a process (a testing process) that offers a visitor or user an original version (A) or a variant (B), at random, in order to compare performance and determine the most effective version for a given audience.

But for it to work, there is one golden rule: do not mix everything up.

The most common A/B testing mistake: testing everything at once

On paper, it is tempting: “Let’s change the title, the button, the layout, the image, the price… and see what happens.” In practice, it is the best way to understand nothing about the result.

Why? Because you would not know which element actually had an impact. A clean test is:

• a clear hypothesis
• one main variable tested
• one big change at a time (or else you move to a multivariate test, and that is another story)

Tip: keep your tests “simple” until you have a large number of visitors and stable traffic. Otherwise you will burn time, and your team will end up analysing numbers that explain nothing.

A good A/B test = a hypothesis + one main variable

1) Define the goal and the indicator to track
Before “testing”, you need to define the goal. It could be:

• increase the conversion rate (purchase, lead, sign-up, quote request)
• reduce the bounce rate
• increase the click-through rate
• improve the time spent on the page
• improve the subscription rate

You then choose a main indicator (conversion rate, sign-up rate, etc.), then secondary metrics for the analysis.

2) Create a clearly identifiable variant
A very handy good practice: give an identifier to each variant (and to version A). Example:

• A_hero_original
• B_hero_image_new
• B_CTA_text_v2

It helps with data collection, reporting and documentation of the internal test write-up (or your report).

3) Make sure the sample is large enough
A test needs a sample (sample size) and a consistent test duration. The more subtle the change you are testing, the more people you need to get statistically significant results.

Examples of elements you can A/B test on a web page

The idea is to choose a single main lever. Here are the main categories of elements you can test on a website, a landing page or a product page.

1) The button (CTA): small change, potentially big impact
The button (call to action) is often one of the best starting points for a test because it is directly tied to the click and to conversion.

You can test:

• the text: “Request a quote” vs “Get my quote”
• the position: higher / lower on the page
• the colour (in moderation)
• the size / style (but be careful not to change 10 parameters)

Concrete example: on a campaign landing page (paid search or social media), a variant with a clearer CTA can improve the conversion rate, without touching the rest of the content.

2) Title, hook, social proof: clarify to convert
The title and the hook answer a simple question: “Why should I stay?”. This is often where the first click of interest happens.

You can test:

• a benefit-led title vs a descriptive title
• a shorter hook vs a more detailed one
• social proof: reviews, client logos, figures (for example “2,000+ clients”)

It influences understanding, trust and therefore the conversion rate.

3) Layout, price, reassurance: reduce the barriers
Here we touch the perceived functioning of the page and the user experience.

You can test:

• a simpler presentation (fewer blocks)
• the order of sections (reassurance before the price, or the reverse)
• a more readable price block
• guarantee / delivery / returns messages
• reassurance elements (secure payment, “no commitment”, etc.)

Goal: reduce the risk perceived by the user and improve the move to action.

4) Media: image, video, carousel
Media can change how the product and the brand are perceived and how well the offer is understood.

You can test:

• a lifestyle image vs a product image
• a short video vs a static visual
• a carousel vs a single image

Be careful: it is often heavier in data (weight, performance), so think about web performance (otherwise you degrade loading time and skew the results).

5) Offer: delivery, bundle, guarantee, promotion
This is a very strong “business” lever, but a riskier one: you change what directly influences margin and perceived value.

You can test:

• free delivery vs a discount
• bundle (pack) vs single product
• a guarantee put forward
• a promotion displayed differently

Tip: on a product page, an offer test can increase conversions, but also check the impact on revenue and profitability (otherwise you win sales but lose profit).

What if you want to test several things at once? The multivariate test (with precautions)

The multivariate test is used to test a combination of changes: for example, 2 titles × 2 images × 2 CTAs = 8 versions. It is powerful, but it requires:

• a large number of visitors
• solid organisation
• a good tool (often a dedicated SaaS platform)
• a good capacity for statistical analysis

Otherwise, you will not get reliable results.

The ABC of A/B testing to keep in mind before every test

Good A/B testing is not a one-off “shot”: it is a structured marketing technique, useful to any company looking to maximise its digital performance. If you keep:

• a clear hypothesis,
• one main change,
• a large enough sample,
• and a clean analysis,

you will get actionable results, avoid rolling out risky changes, and optimise your site in a genuinely effective way.

The step-by-step method for launching a clean A/B test

Define the goal and the primary metric

You need to define how you will judge the test: conversion, click, subscription, bounce, time spent…

• Primary KPI: conversion rate
• Secondary KPIs: conversion rates by device, average basket, bounce rate, time spent, etc.

Tip: choose a KPI tied to the business (for example conversion), not just a “vanity” KPI (for example clicks) if the goal is sales.

Formulate a testable hypothesis

A hypothesis must be actionable:
“If we change X, then Y will increase, because Z.”

Concrete example:

• Hypothesis: “If we use a more explicit CTA on the landing page, the subscription rate will increase, because the intent is clearer.”
• Variable tested: the CTA text on the button.
• Context: marketing campaign (paid search + social media).

Choose the target, the segmentation and the group

You can test on everyone, or on a specific group (mobile only, new visitors, paid search traffic…). The more you segment, the larger the sample size you need.

Build the variants (without breaking the rest)

• Original version: A
• Variant: B (modified page)
• Keep everything else the same: otherwise you are no longer measuring the difference.

Technical tip: give a clear identifier to each variant (for example cta_green_v1, cta_text_v2). It helps the analysis, the code and the team’s reading.

Set up tracking and validate the data

Before launching (pre-launch phase), make sure that:

• The events (the tracked events on the site) are correctly reported in Google Analytics (GA4) and/or Adobe Analytics
• The “conversion” tag is correct (purchase, lead, sign-up)

You can compare A vs B over the same period (test time).

Useful tools:

• Google Analytics, Adobe (Adobe Experience Cloud suite), and testing platforms (Kameleoon, Optimizely)
• Note: Google Optimize existed for a long time as a simple solution, but today the trend is towards alternatives (worth knowing if you come across old “google optimize” tutorials).

Define the test duration and the stopping rules

The test duration depends on traffic and the expected effect. The trap: stopping too early “because it looks better”.

Goal: reach a sufficient volume (sample) to aim for a result known as “statistically significant”.

Statistics without pain: interpreting an A/B testing result

Three possible outcomes:

1. Winner: B improves the primary KPI significantly
2. Loser: B lowers performance
3. Inconclusive: not enough data, effect too small, or… nothing to gain on this element

A few useful notions (without writing a thesis):

• Sample size: the number of users needed
• “Significant” / “most significant”: an indicator of confidence in the observed difference
• Minimum detectable effect: the smallest improvement you want to detect

In practice, the goal is to obtain usable results to decide what to do next: keep the variant, iterate, or go back to the original.

The classic pitfalls (and how to avoid them)

Mistake #1: traffic too low → false positives / false negatives

Mistake #2: testing too many elements at once (unless you are running a multivariate test)

Mistake #3: changing the page in the middle of a test (the test is no longer “clean”)

Mistake #4: poorly chosen KPI (for example CTR up but conversion down)

Mistake #5: tracking / consent / adblock bias

Tip: document each test like a mini internal article: goal, hypothesis, variants, tracking, duration, result, decision.

A/B testing and SEO: friends or enemies?

A/B testing and SEO can coexist, with a few rules:

• Do not do “cloaking” (showing something different to bots)
• Avoid split URL tests without precautions (canonical, consistency)
• Keep a reasonable duration
• Check that the page stays accessible and the content is not broken

3 concrete examples (website, email, app)

Example 1: Landing page (marketing campaign)

• Goal: increase the conversion rate
• Test: CTA “Request a quote” vs “Get my quote within 24 hours”
• Result: better rate on variant B, more qualified leads

Example 2: Email

• Goal: increase the click-through rate
• Test: subject line + preview text
• Result: +X% clicks, but the conversion behind it still needs analysing (do not stop at the click)

Example 3: Mobile app

• Goal: improve sign-up (subscription rate)
• Test: 2 screens vs 1 screen (friction)
• Result: lower bounce rate, higher completion

Building a test roadmap that pays off

A real experimentation strategy (experimentation process / “testing process”) follows a cycle:

1. data gathering (heatmaps, analytics, feedback)
2. test ideas
3. prioritisation
4. test
5. analysis
6. rollout / iteration

This is where an A/B testing solution (SaaS) such as Kameleoon or Optimizely helps: segmentation, feature experimentation, personalisation, dashboards, etc.

A/B testing tools: how to choose

Choosing an A/B testing tool (a testing solution) is not only a question of a “feature catalogue”. The right choice depends on your need, your stack, your traffic volume and, above all, your ability to set up a reliable experimentation process (from the brief to the analysis of results).

A tool can be very powerful on paper… and unusable day to day if the integration is complicated or if the team cannot operate it without friction.

Here are the most important criteria, with a practical, concrete reading.

1) Integration: CMS, stack, performance (and “does it slow my site down?”)
Why it matters: a testing tool directly touches your web pages and the users’ experience. A poor integration can create bugs, visual “flicker” or degrade performance (loading time), which skews the test… and lowers the conversion rate.

What to check:

• Compatibility with your CMS (WordPress, PrestaShop, Shopify, headless…) and your stack (front-end code, framework, tag manager).
• Deployment mode: browser-side script, server integration, SDK.
• Performance impact: loading, rendering, stability.
• Cache handling (CDN, Varnish, static pages): essential to avoid inconsistent versions.
• Ability to handle “simple” changes (text, blocks, CTA) as well as more structural modifications.

Tip: if your site is highly optimised for performance, favour a solution that handles loading well and limits extra layers. Otherwise you may be testing… the impact of the tool, not that of your variant.

2) Targeting: audience, device, source (and smart segmentation)
Why it matters: a global test can hide opposite results depending on the segment. Variant B may be better on mobile but worse on desktop. Or work on SEO traffic, but not on a paid campaign.

What to check:

• Segmentation by device (mobile/desktop/tablet).
• Segmentation by source (SEO, paid search, social media, email, referral).
• Segmentation by user type (new vs returning), geographic area, language, etc.
• Clear targeting rules that are easy to document.

Good habit: if you run a lot of campaigns (paid search landing pages, social ads, emailing), the fineness of targeting becomes a major criterion. You want to isolate “the truth” for a given audience, not a blurry average.

3) Support for multivariate testing and advanced tests
Why it matters: as long as you run simple A/B tests, many tools will do. But as soon as you want to:

• test several variables at the same time (title × image × CTA),
• run tests on journeys,
• or do personalisation,

you need a tool that holds up.

What to check:

• Native A/B/n (more than 2 variants).
• Multivariate testing (true multivariate, not just “several variants”).
• Handling of complexity: number of combinations, governance, readability of results.
• Possibility of “feature experimentation” (tests on features or product rules, not only content).

Tip: multivariate is powerful but needs a large number of visitors and good statistical maturity. The tool should help you avoid “variant madness” while staying manageable.

4) Quality of statistics and reporting (the heart of the matter)
Why it matters: if the stats are opaque or badly calculated, you are making decisions on sand. The tool should help you get reliable results (or conclude “inconclusive” without forcing a winner).

What to check:

• Clarity of reporting: primary/secondary KPIs, segmentation, impact.
• Statistical method (without necessarily going into theory): confidence, stability, handling of variations.
• Conversion handling: attribution, time window, multiple events.
• Export / sharing: PDF, link, dashboard, BI integration.
• “Actionable” reading: recommendations, bias alerts, sample size indications.

Good sign: a good tool helps you decide as much as it helps you not decide too quickly. It should prevent premature stops and hasty conclusions.

5) Ease of set-up for the team (marketers + developers)
Why it matters: most testing programmes fail not because of the technology… but because nobody can operate them day to day.

What to check:

• Test creation interface (for marketers).
• Validation workflow (QA, preview, permissions, approval).
• Collaboration: comments, history, documentation, naming of variants.
• Development needs: “no-code” for simple tests, and an advanced mode for developers.
• Environment management (staging/prod) and versioning.

Tip: if you are a small team, favour a solution that lets you launch simple tests quickly. If you have a mature product/tech team, you can aim higher (feature experimentation, server-side, etc.).

Tools often mentioned in A/B testing: how to position them

Kameleoon and Optimizely
These are dedicated solutions, and these platforms are often mentioned because they generally cover:

• classic A/B testing,
• advanced targeting and segmentation,
• sometimes personalisation,
• and, for some set-ups, product-oriented approaches (feature experimentation).

In practice, they are often chosen when you want to industrialise an experimentation strategy across several pages, several audiences and several teams.

Analytics suites: Google Analytics and Adobe
Google Analytics (GA4) and Adobe Analytics are not strictly “A/B testing tools” (they measure, they analyse), but they become indispensable to:

• track KPIs,
• cross-check results (source, device, segments),
• validate data consistency,
• and build dashboards.

They often serve as the “data foundation”: you test with one tool, but you also check and analyse in your analytics suite.

(Historical) Google Optimize
Google Optimize was very popular because it was simple and accessible, especially with GA. Today it is mostly worth mentioning to understand old tutorials and the history of the market: if someone says “we did that on Optimize”, it refers to a time when a Google-integrated solution existed. For a current project, you would generally turn to other solutions.

Quick guide: choosing the right testing solution for your context

• Simple need + speed (CTA, titles, blocks): favour a solution that is very easy to set up, with good targeting and clear reporting.
• Structured CRO programme (lots of tests, segmentation, governance): a dedicated platform such as Kameleoon/Optimizely + solid analytics.
• Product / features (tests of rules, journeys, features): aim for a “feature experimentation” approach + clean technical integration.

A/B testing in conclusion

A/B testing is a powerful tool in web analytics for a company looking to maximise its web performance: you compare two versions, you measure, you make an informed decision. Done well, A/B testing lets you improve and increase conversion, reduce rollout risk, and offer a more personalised user experience.