You probably blocked the AI shopping agents by accident

Somewhere between 2023 and 2024, every ecommerce team got the same advice. AI companies are scraping your content to train models and giving you nothing back, so block them. It was reasonable advice about training. It was implemented as a broad rule in robots.txt, and that broad rule is now quietly removing stores from the buying set.

Two families of bots with almost the same names

The crawler that collects pages to train a model and the fetcher that reads your page live to answer a shopping question are separate agents run by the same companies under names that differ by a hyphen and a word.

• GPTBot gathers training data. OAI-SearchBot is the one that fetches your page when someone asks ChatGPT to find a product. Blocking one has no effect on the other, and most merchants who intended to block training blocked both.

• ClaudeBot and the user triggered fetchers Anthropic runs are likewise distinct. Share of sites disallowing ClaudeBot roughly doubled through early 2026, from about 4.9 percent to just over 10 percent of disallow rules observed across Cloudflare's network.

• Google-Extended governs Gemini training. It is not Googlebot, and it is not the surface that serves shopping answers in AI Mode. Teams routinely conflate all three.

The scale of the exposure is specific to retail. Analysis of robots files across Cloudflare's network found that retail absorbs around 28 percent of all AI crawler traffic, with the shopping and general merchandise vertical alone accounting for roughly 31 percent of AI crawling activity through 2026. Retail is where these bots go, and retail is where the blanket blocks landed.

The configuration most direct to consumer brands actually want is to block the training crawler and allow the live fetcher. Almost nobody wrote that. They wrote one line that did both.

The second wall nobody edits

Your robots file is a request. Your bot manager is a decision. A store can publish a perfectly permissive robots policy and still return a challenge page to every agent that arrives, because the WAF or bot management layer at Cloudflare, Akamai or Fastly is classifying the request as non human and acting on that classification.

This is the failure mode we see most often, and it is the hardest to notice, because the two layers are owned by different people. Marketing owns robots.txt. Infrastructure owns the bot rules. Neither is looking at what the other did, and the outcome is a store that says yes in writing and no in practice.

Nothing reports it. A blocked agent is not a lost session in your analytics, because your analytics runs in JavaScript that the agent never executed. There is no bounce, no error, no line in a dashboard. The revenue simply goes to a competitor whose page returned a 200.

How to check in ten minutes

1. Open yourstore.com/robots.txt and read it as a policy document rather than a config file. Look for a broad Disallow: / under any AI user agent, and for any named agent you cannot immediately classify as training or live fetching.

2. Request a product page with no browser attached. A plain curl -A "OAI-SearchBot" against a product URL should return your real HTML with a status of 200. If you get a challenge page, an interstitial or a 403, the bot wall is overriding your stated policy.

3. Search the returned HTML for your price. Not the page in your browser, the page curl gave you. If the number is not there, an agent cannot see it either, no matter how permissive your robots file is.

4. Check the bot manager rules directly. Verified agent traffic should be allowlisted by published IP range and reverse DNS, never by user agent string, which anyone can spoof and which therefore gets blocked as a class.

What good looks like

A deliberate policy, written per named agent, with a comment beside each line explaining which decision it encodes. Training crawlers get whatever answer your business actually wants. Live shopping fetchers get allowed, because that is a distribution channel and you would not block a price comparison site either.

Then the same decision gets mirrored at the bot manager, allowlisted properly, and tested from outside with an unauthenticated request. The test is the part people skip, and it is the only part that proves anything. A robots file is a statement of intent. A 200 with your price in the body is evidence.

Sources

• Analysis of robots.txt AI crawler rules across Cloudflare's network, Q1 and Q2 2026

• OAI-SearchBot user agent and blocking rules reference

• Adobe: AI traffic grows but retail sites lag in AI search visibility