Your content only enters ChatGPT, Claude, or Perplexity if the crawler behind each model can reach it first. Robots.txt is the door that either lets them in or locks them out — and in our audits, most sites never check which side of the door they're on.
We run GEO audits on live websites every week. Between June and September 2026, we read robots.txt files from 25+ well-known sites — marketplaces, hotel chains, universities, travel booking platforms. This guide combines what those files actually contain with a practical setup you can copy.
The 14 AI Crawlers That Matter (Updated List)
New AI crawlers appear faster than most blogs can track. These are the ones our crawler database tracks, grouped by what they feed:
| Crawler | Owner | What it feeds |
| GPTBot | OpenAI | ChatGPT knowledge base |
| OAI-SearchBot | OpenAI | ChatGPT search |
| ChatGPT-User | OpenAI | ChatGPT browsing by logged-in users |
| ClaudeBot | Anthropic | Claude knowledge base |
| anthropic-ai | Anthropic | Claude via API / batch |
| PerplexityBot | Perplexity | Perplexity answers |
| Google-Extended | Google | Gemini + AI training data |
| meta-externalagent | Meta | Meta AI assistant |
| CCBot | Common Crawl | Public training archive |
| Bytespider | ByteDance | TikTok / Doubao |
| cohere-ai | Cohere | Enterprise AI search |
| Diffbot | Diffbot | Knowledge graph extraction |
| OAI-GPTBot-XR | OpenAI | XR devices |
| Chrome-ICL | Google | Chrome AI features |
Blocking the first five kills most of your AI visibility in one move. That's why we recommend checking them before anything else.
What 25+ Major Sites Actually Do (Our Audit Data)
Theory is cheap. Here's what we found reading robots.txt files with a plain curl request, no browser:
The "replicate Googlebot" approach — Airbnb (audited Aug 2026). Airbnb's robots.txt gives 7 AI bots (GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, anthropic-ai, cohere-ai, meta-externalagent) the same rules as Googlebot, then adds a 248-line block list for GPTBot that only allows /calendar/ical/ and AMP key paths. The signal: they want AI crawlers to see the site, but not their dynamic reservation pages.
The surgical approach — Coursera (audited Aug 2026). Coursera allows GPTBot, ClaudeBot, PerplexityBot and Google-Extended globally, then carves out exceptions: Disallow: /lecture/ for all of them, meta-externalagent blocked site-wide, OAI-SearchBot blocked from /api/. This keeps lecture content out while letting lesson overviews in.
The hard blocker — TripAdvisor (audited Aug 2026). TripAdvisor explicitly blocks Google-Extended and meta-externalagent with Disallow: /. Ironically, the site itself returns 403 to plain text requests, so even allowed crawlers struggle to read it.
The accidental offenders. These surprise us the most:
- Etsy (Aug 2026): 1,818-line robots.txt with zero AI crawler entries — and the homepage 403s plain-text clients anyway.
- Chipotle (Sep 2026): the robots.txt path itself returns a 404 error page.
- Marriott (Aug 2026): robots.txt is behind Akamai "Access Denied" — the file, not a page, is unreachable.
- Hilton, Hyatt, IHG (Sep 2026): all three return 403 to plain-text requests, no bot negotiation possible.
- Five hospitality chains with readable robots.txt files (Sep 2026): every single one had zero AI crawler rules — they still treat robots.txt as a Google-only concern.
Not one of the 25+ sites we audited blocks AI crawlers by mistake in a way that's easy to spot from the homepage. The misconfigurations hide in missing rules, 403s, and bot walls — which is exactly why a per-bot check beats eyeballing the file.
Should You Allow Everything or Manage Access?
Two workable schools, and both are defensible:
Allow everything. If your content is public and you want maximum AI visibility, give every crawler Allow: / and move on. Risk: AI assistants may surface internal notes, pricing experiments, or thin pages you didn't intend as answers. Google-Extended is the one you might still separate, because it feeds Gemini directly.
Manage per-bot. Follow Coursera's pattern: allow the knowledge-base crawlers broadly, block the paths that shouldn't be quoted (/lecture/, /api/, /dashboard/, search results pages), and keep an eye on what actually gets cited. This keeps control without vanishing from AI answers.
Our recommendation: start with allow-everything, then tighten only the paths that show up in citations you don't want. Blocking everything "until we figure out AI" costs you visibility you can't get back quickly — crawl frequency for new bots is grant-based and slow to build.
The Setup Template
Here's the working baseline we ship in audits (adjust the blocked paths to your site):
* (all others) | Allow: / | Never use Disallow: / on the catch-all — it blocks AI crawlers too |
| GPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, CCBot, Bytespider | Allow: / | The knowledge-base core |
| All of the above | Disallow: /api/, /dashboard/, /search, /cart | Paths that shouldn't be quoted |
| ChatGPT-User | Disallow: / | Optional: stops logged-in browsing from burning tokens on private areas |
Then verify each bot actually resolves to a working fetch, because a robots.txt that returns 404 or a bot wall does nothing. Our free
AI Crawler Check tests all 14 bots against your robots.txt and reports blocked vs allowed in one run.
Four Things That Actually Break AI Crawling
Disallow: / under * — the catch-all that silently kills every AI bot. The most common mistake we see in audits.- Robots.txt behind protection — Marriott's Akamai wall and Chipotle's 404 mean crawlers can't even read the rules. Test
curl -I https://yoursite.com/robots.txt with a plain user agent.
- No rules at all for new bots — meta-externalagent, cohere-ai, and Diffbot appeared in 2024-2025; if your file predates them, they default to allowed, but you never confirmed it.
- Assuming Google = no AI risk — Google-Extended is separate from Googlebot. Blocking it doesn't touch your ranking, so many sites leave the block on and wonder why Gemini never cites them.
Add an
llms.txt file after you fix robots.txt — it tells the crawlers which pages deserve extraction, which is the difference between being crawlable and being cited.
Run the AI Crawler Check on your domain now — it takes one URL and reports all 14 bots plus the exact rule that blocks each one.
