Notes on measurement

How causal measurement actually works, what separates a defensible result from a flattering one, and where the common designs fall down.

Written for marketing and finance leaders who have to defend a number.

• Funny Numbers: self-published marketing results and prevalence of undisclosed methodology

  6,815 marketing case studies from 369 firms. The agencies that demonstrably run controlled designs withhold the method on 76.6% of their own published claims. Paper and summary, both free.

• The marketing measurement checklist

  The five conditions a result needs before anyone can read it, settled at the design stage rather than audited afterward. One worked example runs through all five and takes a 31% claim down to the 3.4 points a holdout can support. Page and PDF, both free.

• Why a media mix model needs an experiment

  A model fitted to observation alone returns confident numbers regardless. The priors and the seasonality controls can quietly do the work, and nothing inside the model says so.

• Why your platforms disagree with each other

  Attribution windows, view-through and identity resolution. Every platform answers a different question, and the total is always too big.

• Last-click is a reporting convention, not a measurement method

  It was built to balance a ledger, not to establish cause. The bias runs in one direction: toward the channels that intercept demand, away from the ones that create it.

• Matched markets or synthetic control

  Matched-market testing is the standard approach. Synthetic control is the stronger one. The difference is in how the comparison group gets built, and whether you can show your work.

• Alternatives to geo testing

  Ghost ads, platform lift studies and matched panel. What each can honestly claim, sorted by who holds the randomization: you, the platform, or nobody.

• Observed incrementality

  When no experiment was run: the 2x2 difference-in-differences design, time-based holdouts, and the assumption both of them rest on.

• What makes a geo test result defensible

  A correctly designed test can still produce nothing worth acting on. Degrees of freedom, statistical power, and what has to be written down before the data arrives.