Common questions
Incrementality testing, in plain terms
What is incrementality testing?
Incrementality testing measures the sales a channel actually caused, by deliberately changing the spend somewhere and comparing what happened against what would have happened otherwise. It is an experiment rather than an attribution rule, which is why it can settle an argument that a dashboard cannot.
What is incrementality, and how is it different from ROAS?
Incrementality is the revenue that would not have happened without the ads. Platform ROAS counts every conversion the ad touched, including the sales you would have won anyway, so it is almost always the larger number. Incremental ROAS is the honest version of the same ratio, and the gap between the two is what a test measures.
What is a geo holdout test?
The most common way to run an incrementality test on paid media. You keep a channel running in most of your markets and turn it off, or hold it flat, in a set of comparable ones. The difference between the held-out markets and what they would have done otherwise is the effect the channel caused.
How many markets do I need to run a geo test?
More than most people expect, and the number is set by how confident you want to be rather than by your budget. Below roughly a dozen separately observable markets, a geo test cannot produce a significance claim at any effect size. That floor is structural, and more spend does not move it. The tool above tells you where your footprint lands.
What is a minimum detectable effect, and how do I calculate it?
The minimum detectable effect, or MDE, is the smallest lift a test could reliably tell apart from noise. It is not a formula you apply to a footprint. It depends on how much of your week-to-week movement is shared across your markets, which only shows up once your own history is run through the design. Anything that returns a single number from a form is guessing.
How long should an incrementality test run?
Long enough to read the effect and short enough that nothing else lands inside the window. Longer tests do read smaller effects, but far less than proportionally, and every extra week is another chance a promotion, a competitor move or a seasonal break contaminates the result. We prefer shorter tests that read as a single point in time.
What should I look for in an incrementality testing agency?
Ask who benefits from the answer. An agency paid on the media it buys is the wrong party to ask whether that media worked. Ask what the test cannot claim, in writing, before it runs. And ask how the method behaves on data where the true answer is already known to be nothing, because a method that finds effects there will find them in your data too.
Do I need an incrementality testing tool or platform?
A tool runs the arithmetic. It does not tell you whether your footprint can support the test, which design fits your business, or what the result is allowed to claim. Most of the expensive mistakes here happen before any arithmetic runs, which is why the free tool above answers feasibility rather than handing you a number.
Can you design and run an incrementality test for us?
Yes, and it is most of what we do. We size the test before you spend anything, design the market split, set in writing what the result can and cannot claim, and read it when it is done. We do not buy or execute media, so we have nothing to gain from the answer coming back favorable. Trevor is at 775.440.3976.
Are you confident you can design, run and measure a geo test?
Yes, and the useful version of that answer is the limit rather than the claim. Every design we run is first checked against data where the true answer is known to be nothing, and if it reports a finding there we do not ship it. That is why some prospects hear from us that their footprint cannot support the test they were about to buy.
Is an incrementality test better than media mix modeling?
They answer different questions. A model describes what has already happened across every channel at once and needs no new spend. A test creates evidence about one channel by changing something on purpose. A model is worth far more once it has an experiment to calibrate against, which is the case for running both rather than choosing.
Can I run a geo test if I sell through retailers or a marketplace?
Usually not. A geo test compares treated markets against untreated ones, so you need revenue you can already see market by market in your own system. If sales arrive through wholesale or a marketplace, there is nothing to compare, and the answer is a different design rather than a smaller one.
Does an incrementality test need a big budget?
It needs spend that is a meaningful share of the revenue in the markets you test, which is not the same thing. A large national budget spread thinly can be harder to measure than a smaller one concentrated in fewer places, because what decides it is the size of the effect against the size of the noise.
There is more of this, at length, in the
notes.
