The method hasn’t changed. The traps have.
An A/B test splits a comparable audience into two arms and compares them on a single metric chosen before launch. Because both versions run at the same time, a gap in results can be tied to the one element you changed – a headline, a landing page, a bid strategy, an asset group. That is still the cleanest way to separate your decision from seasonality, competitor moves and audience mix.
What has changed is where tests live. Google, Meta and Microsoft now run experiments inside the buying platform, and the results can flow straight back into live campaigns. Google Ads turns on auto-apply by default for new experiments, so a “test” can rewrite your original campaign unless you switch it off.
Where winners quietly mislead
Three things turn a declared winner into noise:
- Peeking. Checking results and stopping at the first sign of significance breaks standard p-values. Researchers proposed “always valid” measures for exactly this habit.
- Divergent delivery. A platform may show each variant to different audience segments, so the gap reflects who saw it, not what it said.
- Relative, not incremental. Comparing two versions tells you which one did better on the traffic they got. It does not prove either version caused a sale that would not have happened anyway.
Time is the other variable. Google’s guidance points to at least four to six weeks, longer when conversions lag, and it excludes the first seven days of ramp-up data.
Run the next one like a decision, not a demo
Start with a hypothesis and one primary metric. Pick a cookie-based split where possible, give the test 50% of traffic, and set your confidence level before you open the results. If you want a causal read, a holdout design is the stronger tool – Google’s incrementality experiments now start at $5,000, down from a level near $100,000. In January 2026, Performance Max asset tests added unequal splits such as 80/20, useful when you’re confident in the control but still want a safer read.
Cross-campaign budget and ROI target tests are next: Google described them in August 2026, with rollout beginning in September. That matters because budget-level changes have been hard to test without duplicating campaigns.
The test is not the strategy. It’s the mechanism that tells you whether a creative, bid or budget change is worth scaling. If you peek, let the platform auto-apply, or skip the holdout when you need causality, you’ll end up with a winner that doesn’t hold up in the next quarter.
Source: PPC Land
