Incrementality testing
Also known as: Conversion lift testing, Holdout testing
In short
A controlled measurement approach that estimates the additional outcomes caused by an intervention compared with what would have happened without it.
Measure the difference the advertising causes
Incrementality testing asks whether advertising produces additional outcomes, beyond those that would happen anyway. A test compares a treatment group exposed to the intervention with a suitable control group.
The groups must support a fair comparison. Random assignment helps reduce systematic differences; geographic experiments require an appropriate design and adjustment. The outcome definition, test period and analysis plan need to be established before interpreting results.
Attribution answers a different question: how observed conversions receive credit across interactions. An attributed purchase is not automatically an incremental purchase.
Illustrative calculation
Imagine a fictional randomized test with equally sized groups of 10,000 eligible people. During the same period, the treatment group records 240 purchases and the control group records 200.
The observed difference is 40 purchases. Relative to the control group’s 200 purchases, the observed lift is 40 ÷ 200 × 100 = 20%.
Those figures illustrate the calculation only. They do not establish statistical reliability, and unequal group sizes would require normalization before comparing totals.
Keep uncertainty visible
Check sample size, confidence or credible intervals, conversion delays and contamination between groups. A test can be inconclusive. A result for one audience, budget and period does not establish the effect of every future campaign or every additional unit of spend.