Robots.txt is the first place most stores set an AI policy, but it is a request, not a wall. The useful move for a shop is to treat AI bots as three groups: training crawlers, search crawlers and agents acting for a shopper. Blocking the first group costs you nothing in AI shopping. Blocking the second and third can quietly remove you from ChatGPT, Gemini and other agents' answers. This guide shows the three policies stores use, with sample rules, as of November 2026.
- AI agents disallowed by amazon.co.uk/robots.txt (10 Nov 2026)
- 99
- OpenAI user agents with published IP lists
- 4
- Time OpenAI says search opt-out changes take
- ~24 hrs
- Legal status of robots.txt rules
- Advisory
Which AI bots visit a store, and what does each do?
The crawlers worth naming fall into three groups.
| Group | Examples | What it does | Shopping impact of blocking |
|---|---|---|---|
| Training | GPTBot, Google-Extended, ClaudeBot | Collects content that may train models | None on shopping answers |
| Search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Indexes pages for AI search answers | You can drop out of AI results |
| User-triggered agents | ChatGPT-User, Claude-User, Perplexity-User, GoogleAgent-Shopping | Fetches a page because a person asked | Agents cannot check your page for that shopper |
OpenAI's crawler documentation is the clearest vendor statement. GPTBot crawls content that may be used to train foundation models. OAI-SearchBot surfaces sites in ChatGPT search, and opted-out sites "will not be shown in ChatGPT search answers"; changes take about 24 hours. ChatGPT-User visits pages when a user asks ChatGPT something and "is not used for crawling the web in an automatic fashion". Because a user initiates those visits, OpenAI says "robots.txt rules may not apply". OpenAI publishes IP lists for each bot.
Claude Duet's browser sends its own User-Agent header and Anthropic publishes the IPs it browses from, according to Anthropic's Duet safety article. Some agents, such as Meta Muse, have no dedicated user agent that a robots.txt line could name.
Why is robots.txt contested for shopping agents?
There are two positions, and both have serious backers.
- The site owner's view: robots.txt and terms of service express what automated access a site allows, whoever triggers it. Amazon's Agent Terms, published 30 May 2025, require shopping agents to identify themselves as "Agent/[name]".
- The agent vendor's view: an agent fetching one page because a person asked is acting like that person's browser, not crawling. OpenAI's statement about ChatGPT-User reflects this framing.
This disagreement sits at the centre of Amazon's lawsuit against Perplexity over the Comet browser agent, in which a federal judge granted Amazon a preliminary injunction on 9 March 2026. Read Amazon v. Perplexity, explained and Amazon's agent terms. When Amazon blocked Muse on 20 September 2026, Search Engine Journal noted that robots.txt played no part, because Muse did not announce itself.
Policy 1: Allow everything
Most small and mid-size stores benefit from being found and bought from. If that is you, an open policy is simplest:
User-agent: *
Disallow: /cart
Disallow: /account
Allow: /
Keeping cart and account paths out of crawlers is ordinary hygiene; user-triggered agents completing a purchase will still use them when a shopper asks.
Policy 2: Block training, allow shopping
The most common middle ground: opt out of model training while staying visible in AI search and usable by agents.
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: *
Allow: /
Google-Extended is a control token rather than a separate crawler, so it affects use of your content in Google's AI models, not Google Search or Shopping indexing. Check each vendor's current documentation before copying names, because user-agent lists change.
Policy 3: Block AI agents (the Amazon approach)
As checked on 10 November 2026, amazon.co.uk/robots.txt disallows 99 named user agents. They include ChatGPT-User, GPTBot, OAI-SearchBot, ClaudeBot, Claude-User, Claude-SearchBot, GoogleAgent-Mariner, GoogleAgent-Shopping, Google-Extended, Copilot, Perplexity-User, PerplexityBot, meta-externalagent, xAI-Grok, GrokBot and Gemini-Deep-Research. A blocking policy looks like this:
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: GoogleAgent-Shopping
Disallow: /
Very large retailers can afford this because shoppers come to them directly, and they want agents to go through official channels such as their own assistants or negotiated partnerships. Most stores cannot. See which stores allow AI agents for how major retailers compare.
Pros
- Allow all: maximum discovery and agent purchases
- Block training only: keeps AI visibility, limits content reuse
- Block agents: control over who transacts and how
Cons
- Allow all: no say over how content trains models
- Block training only: relies on vendors honouring the split
- Block agents: lost sales and no technical guarantee
How do you enforce a policy for real?
- Request
- Check signature or published IP
- Match to policy
- Allow, challenge or block
Because robots.txt is voluntary and user-agent strings can be faked, enforcement needs something stronger:
- Published IP ranges. OpenAI publishes JSON lists for each bot, and Anthropic publishes Duet's browsing IPs. Check claimed user agents against them.
- Signed requests. Web Bot Auth lets agents cryptographically sign requests. Cloudflare's signed agents category, which started with ChatGPT agent in August 2025, is built on it. See Web Bot Auth for stores.
- Charge instead of block. Cloudflare's pay per crawl, a private beta announced 1 July 2025, uses HTTP 402 responses to charge crawlers per request. It is aimed at publishers more than shops.
Before you block, measure. Our guide to tracking AI agent traffic shows how to see which agents already visit and whether they lead to orders.
Bottom line
For most stores, block training crawlers if you want to, keep search crawlers and user-triggered agents allowed, and enforce anything that matters with signatures and bot management rather than robots.txt alone. Write the file as a business decision and review it quarterly; the list of agent user agents grows every few months. The wider checklist is in making a store agent-ready.


