Marketers often treat every number in an ads dashboard as the same kind of truth. Karan Dhir, principal AI product manager at Genentech and author of a Substack essay called Facebook Proved Its Own Ad Measurement Wrong, says that habit is expensive. He separates platform output into two classes: randomized lift studies, which count as causal evidence within scope, and modeled conversions or algorithmic lift estimates, which he calls operational telemetry.
Facebook’s own experiments set the benchmark
The essay leans on two Marketing Science papers built on Meta’s experimentation infrastructure. The first, from 2019, used 15 US experiments, 500 million user-experiment observations and 1.6 billion ad impressions. Observational methods such as matching and regression often failed to recover randomized results; they frequently overestimated effects and sometimes underestimated them. Dhir says the misses were “by a factor of 3 or more,” though the published abstract gives no single multiplier.
The 2023 follow-up makes the gap concrete. Across 663 large-scale Facebook experiments, median randomized lift was 29% for upper-funnel outcomes, 18% for mid-funnel and 5% for lower-funnel. A machine-learning estimator returned 83%, 58% and 24%. Propensity score matching returned 173%, 176% and 64%. Neither came close enough to be reliable, even with more than 5,000 user-level features.
Why the confusion is growing
Modeled conversions are not new. Google introduced them in Analytics in 2017 and in Google Ads in 2019, typically adding a low double-digit percentage. Under Consent Mode the threshold is now 700 ad clicks in seven days, with audits reporting recovery rates of 15% to 40%. Meta removed 7-day and 28-day view-through windows from the Ads Insights API in January 2026 and, in March 2026, narrowed click attribution to link clicks only.
Privacy changes removed observed conversions, and platforms filled the gap with estimates. Dhir’s worry is that a randomized study can now have an outcome metric partly built from modeled conversions, leaving the number in a category the dashboard does not label.
There is also a structural conflict: the same platform sells the media and grades it. Meta’s own May 2025 white paper put randomized experiments above marketing mix models and rules-based attribution. It also reported, from 307 studies across 54 advertisers, that advertisers undervalued Meta by a median 31% under rules-based models. That is a vendor-produced calibration for its own inventory, which is exactly why Dhir wants independent custody.
Classify before you optimize
Dhir’s practical fix is intake classification. Treat every platform number the way a bank treats collateral: label it before it enters the decision process. His three questions:
- Was assignment randomized, and by what mechanism? A genuine split-before-auction answer is yes; methodology language about adjustment is not.
- Who holds the holdout, and can it be audited? Strong setups include co-specified designs, third-party measurement and pre-agreed statistical power.
- Does the outcome reconcile to your financials? If a lift readout cannot be walked, even approximately, to the ledger, it is not evidence for budget decisions.
Dhir’s standard: randomized, adequately powered and reconcilable results can calibrate models. Everything else is useful for running campaigns but inadmissible for judging them. Recent data points reinforce that. A Zalando researcher’s synthetic-data paper posted to arXiv in August 2026 found a conventional MMM reported ROAS of 10.61 against a true 4.20, while a structural approach using four geo-experiments recovered 4.14. Earlier, eBay’s brand keyword experiments found near-zero incremental effect for spend that attributed reporting had valued strongly.
Dhir leaves little ambiguity for teams still optimizing on modeled lift. “Organizations still allocating on modeled lift aren’t uninformed. They’re unwilling,” he wrote on LinkedIn. The turnaround step is to ask the three intake questions before a number moves a budget.
Source: PPC Land
