SEO teams test everything now: titles, internal linking, content templates, and increasingly the impact of AI-generated changes. But running an experiment and actually learning from it are different skills. A metric can move while you still have little proof that your change caused it; a flat line can hide a real effect you were not positioned to see.
A new Search Engine Journal piece argues SEO should borrow from science. The goal is not a prettier before-and-after chart. It is making it harder to fool yourself about what the result means.
Begin with a hypothesis that can fail
Good experiments begin with a business question, not a generic ambition. “Improve page titles to improve SEO performance” is not testable because “improve” and “performance” are undefined. Instead, make a specific, falsifiable prediction, such as updating titles on 700 pages to include a country reference will generate 10% more clicks over four weeks compared with the previous four weeks.
Before launching, ask: what result would prove this wrong? If no result could, you are collecting confirmation, not evidence.
Track the noise, not just the change
Websites are not sterile labs. While you test titles, product may ship a feature, marketing may launch a campaign, seasonality may shift demand, and Google may roll out an update. Each is a confounding variable that offers an alternative explanation for whatever you observe.
Keep a log of major changes across SEO, engineering, product, and marketing. The author describes scrapping a content-origin study because the materials differed in messaging and familiarity, which would have made the origin impossible to isolate. That discipline hurts in the moment and saves you from confident wrong conclusions.
Lock your success criteria before the data arrives
Confirmation bias is the classic trap: traffic up means your work, traffic down means an algorithm update. To counter it, commit before launch to your primary metric, comparison period, and evidence bar. Then do not quietly switch to impressions, CTR, or new keywords if the main number disappoints.
A simple pre-test checklist helps:
- What is the one primary outcome?
- What comparison makes it meaningful?
- What evidence would support the hypothesis, and what would falsify it?
- Which other changes could explain the result?
Know what your test can and cannot say
There are three possible endings: evidence for your prediction, evidence against it, and not enough evidence to decide. They are not interchangeable. If a test is too short, the sample too small, or the effect too subtle, you may simply lack the data to detect it.
As the piece puts it,
“Absence of evidence isn’t necessarily evidence of absence.”
More data will not rescue a badly designed test, but it can help a well-designed one deliver an answer.
Build a knowledge portfolio, not a single verdict
One test shows what happened with a particular sample, method, and period. It does not prove Google prefers a template or that you should apply a win across every market and page type. Document every experiment: question, hypothesis, treatment, sample, dates, context, result, limitations, and next action.
Twenty documented tests start to reveal which assumptions survive contact with reality.
Only test when the answer changes a decision
Before you run anything, finish the sentence: “Depending on what we find, we will…” If every outcome leads to the same action, the test is busywork. A 0.3% organic click lift that requires six months of engineering may be statistically convincing and commercially irrelevant. Value comes from learning and action, not experiment volume.
Source: Search Engine Journal
