A/B testing
Also known as: Split testing
In short
A controlled experiment that compares alternatives by assigning eligible participants to different versions and measuring a predefined outcome.
Compare alternatives under the same conditions
A/B testing compares an original experience with a variation. Random assignment helps create comparable groups, while simultaneous measurement reduces differences caused by timing.
Define the primary outcome, eligible audience, allocation and analysis approach before starting. A test of headline wording needs a clear decision criterion, such as completed purchases per eligible visitor. Clicks may be useful diagnostics without being the main success measure.
Keep assignment consistent where the experience requires it. A person repeatedly switching versions can make the exposure and outcome harder to interpret.
Illustrative comparison
Imagine a fictional store testing two delivery-information layouts during the same two-week period. Version A receives 5,000 eligible visitors and 150 purchases. Version B receives 5,000 eligible visitors and 170 purchases.
The observed purchase rates are 3% and 3.4%. The difference is 0.4 percentage points, or about 13.3% relative to A. Those arithmetic results alone do not establish statistical significance.
Do not declare a winner too early
Follow the planned analysis and examine uncertainty, sample size and any unexpected imbalance. Repeatedly stopping when a preferred result appears can distort conclusions. A test may be inconclusive. Its result also applies to the tested audience and conditions; it does not guarantee the same effect after a major change in traffic or assortment.