Notes on measurement

How causal measurement actually works, what separates a defensible result from a flattering one, and where the common designs fall down.

Written for marketing and finance leaders who have to defend a number.

• Matched markets or synthetic control

  Matched-market testing is the standard approach. Synthetic control is the stronger one. The difference is in how the comparison group gets built, and whether you can show your work.

• Alternatives to geo testing

  Ghost ads, platform lift studies and matched panel. What each can honestly claim, sorted by who holds the randomisation: you, the platform, or nobody.

• Observed incrementality

  When no experiment was run: the 2x2 difference-in-differences design, time-based holdouts, and the assumption both of them rest on.

• What makes a geo test result defensible

  A correctly designed test can still produce nothing worth acting on. Degrees of freedom, statistical power, and what has to be written down before the data arrives.