Every time you hand Google a tROAS target and walk away, you are handing the keys to a reinforcement learning agent. Same with Meta’s ranking stack. Same with your DSP’s optimisation layer.
PPC Land has published a detailed explainer on how the method actually works, and it is worth every performance marketer’s attention, because the failure modes are as documented as the wins.
The loop, in plain English
Reinforcement learning has no labelled right answers. An agent reads the state of an environment, picks an action, gets a numerical reward, and adjusts the rule it uses next time so long-run reward goes up.
Three things make it different from the supervised models most marketers know. Feedback is evaluative, not instructive: it tells you how good the action was, not what you should have done. Rewards arrive late, so credit has to be traced backwards. And the agent’s own behaviour shapes the data it sees next, which creates a permanent explore-versus-exploit tension.
The plumbing: Markov decision processes, value functions, temporal-difference learning. Q-learning arrived from Chris Watkins in 1989. OpenAI published Proximal Policy Optimisation in July 2017, and it became the default because it stays stable at scale.
Where it sits inside auto-bidding
This is the part most advertisers get wrong. The RL agent is not pricing individual impressions. A separate supervised model predicts click and conversion probability in the milliseconds before the auction closes. The agent turns the dial that scales all of those prices.
Alibaba researchers laid out the standard formulation at ACM SIGKDD in August 2024. The state holds five variables: remaining time, remaining budget, spend speed, real-time cost-efficiency and average cost-efficiency. The action adjusts the bidding multipliers. The reward is value contributed in that period.
Crucially, training happens offline. The same paper notes that learning through direct interaction with the live system is not feasible on safety grounds, so policies get fitted to historical logs and only then deployed. That single constraint explains a lot of what goes wrong later.
Exploration is now a visible dial
Google’s Smart Bidding Exploration turned an internal trade-off into an advertiser control. You accept a lower effective ROAS target so the system will bid on queries it would otherwise skip. Tolerance runs 5% to 30%. Google’s own arithmetic: a 200% target with 10% tolerance gives an effective average target of 180%.
Google reported an 18% increase in unique search query categories producing conversions and a 19% lift in conversions overall, on internal data from 11 March to 11 April 2025. It expanded the feature to Performance Max and Shopping on 15 June 2026.
Meta’s numbers point the same way. Finance chief Susan Li disclosed that user-understanding work plus the GEM ranking system produced an 8.3% rise in ad clicks and 15.7% uplift in conversions on Facebook in Q2 2026, up from a 3.5% click lift two quarters earlier.
What breaks
- Reward misspecification. The agent optimises the reward you gave it, not the outcome you meant. An MIT paper submitted to arXiv on 22 February 2026 traced chatbot sycophancy straight to RLHF, since agreeable answers earn good ratings.
- The offline-to-online gap. Policies trained on logged data face a live market where competitors have already moved. Alibaba named this the inconsistency between online and offline in its 2022 NeurIPS work.
- Contested formalism. The same group argued in 2024 that the Markov assumption does not hold in auto-bidding, and its diffusion-based alternative reported a 2.81% GMV gain over RL baselines.
- Weak evidence. The Trade Desk claimed a 32% CPA cut across 62 campaigns in August 2026 by comparing two model versions, not against a control, with no methodology document.
What to actually do
Treat your conversion action as the reward function, because that is what it is. If it is a soft signal, the agent will farm it. Budget deliberately for exploration rather than pretending your account is at steady state. And when a platform reports a lift, ask whether it was model-versus-model or a real controlled test.
Barto and Sutton received the 2024 ACM Turing Award for founding this field. Their algorithms are now spending your media budget. Learn the vocabulary.
Source: PPC Land
