Website Split Testing: When an A/B Test Is Worth Running
TL;DR
In this guide
- Start with a specific problem. Decide what behaviour the change should improve.
- Plan before starting. Agree the audience, measure and decision rule.
- Use an alternative when needed. Low traffic or broken measurement can make a test unhelpful.
Website A/B testing compares different versions of a page or element for a defined audience and outcome. It can help assess a design decision when traffic, tracking and the experiment setup support a meaningful comparison. It is not the first answer to every weak website.
The guide at a glance
Decide whether a test is the right tool
01
Clarify the hypothesis
Connect one proposed change to a reasoned visitor problem.
02
Check measurement
Confirm exposure, completed actions and data quality can be recorded.
03
Choose a practical method
Use an experiment, a usability review or a direct correction to suit the problem.
Write a test hypothesis before choosing a tool
Describe the page, audience, proposed change and result you expect. A useful hypothesis explains why the change could help. If the enquiry button follows the service explanation, visitors may find the next step more easily. That is a testable idea; “make the page more modern” is not a clear experiment.
Choose a meaningful primary outcome such as completed, relevant enquiries. A button click can be a supporting signal, but it does not show whether the visitor successfully submitted the form or whether the business received a useful lead.
An illustrative test plan
- Question: does a clearer next-step explanation help service-page visitors complete an enquiry?
- Version A: the current page.
- Version B: the same offer with a clearer enquiry prompt.
- Primary outcome: successful form completion for the defined audience.
- Checks: form errors, device behaviour, tracking and enquiry quality.
- Decision: agree the sample, evaluation method and practical threshold before launch.
This plan is an example, not a prediction of uplift. The expected effect, baseline conversion rate and required confidence affect how much data is needed. Ask the analyst or testing provider to justify the method rather than assuming a fixed visitor count fits every website.
Avoid reading ordinary variation as a result
Keep the audience and measurement consistent, avoid changing several unrelated elements without a clear design, and record events that affect the comparison. A campaign, seasonal change or broken form can alter results independently of the tested layout.
Do not stop simply because one version looks ahead early. Use the evaluation approach agreed before the test and inspect data quality. Also check whether a statistically supported difference is large enough to matter to the business.
What to do on a low-traffic website
If the page receives too little suitable traffic for a useful test, begin with direct evidence: review customer questions, inspect form failures, observe representative people attempting the task and check basic readability and mobile use. Correct an obvious broken link or inaccessible form without delaying it for an experiment.
Use the findings to create a better brief for a design change. Review the change honestly, recognising that a simple before-and-after comparison cannot isolate every external influence.
Select a current testing setup
Choose tools around the experiment, privacy requirements, website platform and measurement plan. Check compatibility and costs before implementation. Google Optimize is no longer an available option; Google records its September 2023 closure.
A testing tool does not remove the need for a clear hypothesis, reliable tracking and practical interpretation. Set up the full task and receiving report before claiming a measured result.
Put it into practice
Prepare before running an experiment
- The page, audience and problem being investigated
- One clear hypothesis and proposed variation
- A verified primary outcome and supporting checks
- Baseline data and a justified evaluation method
- A record of campaigns, seasonality and other changes
- Privacy, consent and platform requirements
- A practical decision owner and review record
Keep in mind
More clicks do not always mean better enquiries
Check the complete conversion and the quality of resulting enquiries. A change can encourage more clicks while making the form or decision less useful.
Common mistakes to avoid
- Testing before fixing a broken task or tracking setup
- Choosing the winning version from an early fluctuation
- Changing unrelated page elements without a clear question
- Promising a conversion increase before the experiment
A few useful answers
Common questions
Should a small business always run A/B tests?
No. A practical review, user observation or direct correction may be more useful when traffic or measurement is insufficient.
Can I compare the month before and after a redesign?
You can review the change, but campaigns, seasonality and audience differences can affect the result. A simple before-and-after record does not establish that design alone caused the difference.
