Imagine three shoppers. Same search, same shelf, same instruction. They each reach for a different product. That is what a new agentic-commerce test found when Claude Haiku 4.5, Sonnet 5 and Opus 5 were given the same 20 Shopify catalog results.
The experiment: 1,980 picks, zero shared winners
On Oct. 13, 2026, Shopify opened Shop MCP, a connection that lets shopping agents search its catalog, build carts and check out through Shop Pay without an API key. Testers pulled the top 20 results for six searches—coffee beans, backpack, running shoes, water bottle, wireless earbuds and yoga mat—and asked each Claude model to pick one product 50 times per search.
Across 1,980 decisions, the three models never agreed on the same top product in any of the six searches. Each model was strikingly consistent with itself: Sonnet picked the same product 94% of the time, Opus 89%, and Haiku 67%. Change the model, and a different product won.
What each model optimized for
Opus played it safe. In 222 of 300 picks on the main searches, its reason used the word “safe” or “safest.” It chose general-purpose products and called a medium roast “the safest crowd-pleaser” for a plain coffee request.
Haiku and Sonnet leaned toward review counts. The clearest example: the two models picked a water bottle with a catalog review count of 447,221—even though the product page showed 6,537. Opus ignored that bottle and picked an Owala all 50 times.
Position also mattered, but silently. When the catalog’s top sunscreen was moved from first to last, all three models almost stopped picking it. Across eight skincare searches, mentions of position never appeared in any of the 480 reasons.
When the request changed to “under $25, for oily skin,” every model shifted: 298 of 300 picks landed under $25. A $17.49 sunscreen with a 4.2 rating won a third of Opus’s picks because its listing said “without clogging pores.”
Why this matters for ecommerce teams
Shopify’s catalog feed is becoming a new storefront. The model sees title, store, price, rating, review count, summary and first three features—and it reasons from exactly that. If the feed shows a review count that does not match your product page, the model may buy on bad data.
There is a decision layer between shopper and store that nobody can monitor. A brand cannot choose which model a customer’s agent runs on, and it cannot see the pick happen. But it can influence what the model reads.
Four moves for Shopify sellers
- Audit the agent-facing catalog. Search your product through Shop MCP and check the rating, review count, summary and features the model sees.
- Match listing language to customer needs. “Without clogging pores” beat higher ratings because it matched a stated need.
- Keep claims true. Models repeat “clinically proven” and “11x faster” as their reasons without checking them.
- Test your category across models. Run the same search through a few models and see which product each picks.
The takeaway is not that AI shopping is broken. It is that the shelf itself has changed. When an agent shops, the model is the new gatekeeper—and product feed optimization is the new SEO.
Source: Search Engine Journal
