Do AI crawlers respect robots.txt?

The major AI companies, including OpenAI, Anthropic, Google and Perplexity, say their crawlers follow robots.txt, and they publish their user agents so you can write rules for them. Less reputable scrapers may ignore it or disguise themselves. robots.txt is a request, so anything truly private needs authentication or firewall rules, not a Disallow line.

Will allowing AI crawlers slow down my site?

For a typical marketing site with dozens or a few hundred pages, crawler traffic is small and static or cached pages handle it easily. Problems usually come from aggressive scrapers hitting large sites or uncached dynamic pages. If you see load spikes, rate-limit specific user agents at the CDN rather than blocking all AI crawlers in robots.txt.

Should I block AI crawlers on my staging site?

Yes, but don't rely on robots.txt alone. Put staging behind a password or an IP allowlist and add a noindex header, so neither search engines nor AI crawlers can fetch it. Then make sure the staging robots.txt and noindex settings don't get copied to production at launch, which is a classic cause of a site vanishing from search.

Can I see how often AI crawlers visit my site?

Yes. Your server, host or CDN logs record each request with its user agent, so you can count visits from GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and others. Some CDNs, including Cloudflare, show AI crawler activity in a dashboard. GA4 won't show it, because crawlers don't run the analytics script; it only records human visits.

Does blocking AI crawlers protect my content from being copied?

Only partly. Blocking training bots stops compliant companies from collecting your pages for future training, but it doesn't remove content already collected, and it doesn't stop scrapers that ignore robots.txt. Anything public can be copied by someone. If the content is your product, put it behind a login or a paywall rather than relying on crawler rules.

All posts

Web design & development

Should You Block or Allow AI Crawlers? GPTBot, ClaudeBot and robots.txt

Should you block AI crawlers in robots.txt? A clear verdict for marketing sites, GPTBot, ClaudeBot and PerplexityBot rules to copy, and CDN settings to check.

By Mehdi HassanOct 29, 20267 min read

The short answer

Whether to block AI crawlers in robots.txt depends on what the pages are for. A business should allow AI search crawlers such as OAI-SearchBot, PerplexityBot and Googlebot on its public marketing pages, because blocking them removes you from AI answers. Block private areas, and decide on training bots like GPTBot and ClaudeBot as a separate content-licensing choice.

Key takeaways

Marketing pages exist to be found: allow AI search crawlers on them.
Search bots and training bots are separate user agents, so you can allow one and block the other.
Blocking Google-Extended doesn't remove you from AI Overviews; those use Googlebot.
CDN and firewall bot rules often block AI crawlers even when robots.txt allows them.
robots.txt is a request, not security. Protect private areas with authentication.

Part of How to Build an AI-Search-Ready Website: Structure, Schema and Crawlability

Should you block or allow AI crawlers?

For a company's marketing site, allow them. Your homepage, product pages, pricing, comparisons and blog exist to be found by buyers, and more of those buyers now research in ChatGPT, Perplexity and Google's AI features. If those engines can't fetch your pages, they can't cite you, and they'll quote a competitor or a review site instead.

Blocking makes sense in narrower cases: logged-in app areas, staging sites, internal search results, gated content, and anything you sell as content, like paid research or a publisher's archive. The decision is part of building an AI-search-ready website, and it's one of the few AEO choices that can switch citations off entirely.

Which AI crawlers should you know about?

The AI crawlers that matter split into three jobs: search bots that fetch pages to cite in answers, user-triggered fetchers that load a page when someone asks about it, and training bots that collect content for future models. Each company publishes separate user agents for these, so robots.txt can treat them differently.

User agent	Company	Job	Our default for a marketing site
Googlebot	Google	Search index, which also feeds AI Overviews and AI Mode	Allow
Google-Extended	Google	A control token for Gemini model training and grounding, not Search	Your choice; doesn't affect AI Overviews
OAI-SearchBot	OpenAI	Finds pages to show and cite in ChatGPT search	Allow
ChatGPT-User	OpenAI	Fetches a page when a user's request needs it	Allow
GPTBot	OpenAI	Collects content for model training	Your choice
PerplexityBot	Perplexity	Indexes pages for Perplexity answers	Allow
ClaudeBot	Anthropic	Collects content for model training	Your choice
Claude-SearchBot and Claude-User	Anthropic	Search indexing and user-triggered fetches for Claude	Allow
Bingbot	Microsoft	Bing index, used by Copilot and by other assistants	Allow
CCBot	Common Crawl	Open web archive widely used to train models	Your choice

Does blocking GPTBot remove you from ChatGPT?

No. Blocking GPTBot stops OpenAI from using your pages for model training, but ChatGPT search uses OAI-SearchBot to find pages to cite, and ChatGPT-User to fetch pages during a conversation. If you block only GPTBot, you can still be cited in ChatGPT search answers.

The reverse mistake is more common and more costly: a blanket rule that blocks every AI-sounding bot, copied from a blog post, quietly takes a site out of ChatGPT and Perplexity. The same logic applies to Google. Blocking Google-Extended controls Gemini training and grounding, while AI Overviews come from Googlebot's normal crawl, so the only way to leave AI Overviews through robots.txt is to leave Google Search, which no business wants.

What robots.txt rules should a marketing site use?

A marketing site should use a short robots.txt that allows everything public, blocks private and duplicate paths for all bots, and adds separate groups only for the training bots you choose to refuse. Here's the pattern we use. Each line below is one line of the file; groups are separated by a blank line.

Group	Lines in robots.txt	What it does
All bots	User-agent: *	Starts the default group that applies to every crawler
All bots	Disallow: /app/ and Disallow: /api/ (one per line)	Keeps logged-in and API paths out of every index
All bots	Disallow: /search	Stops crawlers wasting time on internal search results
Optional training opt-out	User-agent: GPTBot, then Disallow: /	Refuses OpenAI training while ChatGPT search still works
Optional training opt-out	User-agent: ClaudeBot, then Disallow: /	Refuses Anthropic training
Optional training opt-out	User-agent: CCBot, then Disallow: /	Refuses Common Crawl
Sitemap	Sitemap: https://yourdomain.com/sitemap.xml	Points every crawler to your full URL list

Note that a crawler follows the most specific group that names it. If you add a group for GPTBot, it ignores the default group, so repeat any shared Disallow lines inside it if you want them applied. For the full basics, see our guide to robots.txt and XML sitemaps and the plain-English definition of robots.txt.

Should you block training bots at all?

It depends on what your content is worth as content. For most startups, the website is marketing: you want the world, including future models, to know what you do, what you charge and who you serve. Letting training bots in means models are more likely to describe you accurately even when they don't search.

For publishers, research firms, course creators and anyone whose pages are the product, blocking training bots is a reasonable licensing stance. Just block them by name and leave the search bots alone, so you keep the citations and referral traffic while refusing free training data.

Why are AI crawlers blocked even when robots.txt allows them?

Because robots.txt isn't the only gate. CDNs, firewalls and hosting platforms have their own bot-management rules, and some block AI crawlers by default or challenge them with a JavaScript check they can't pass. The bot gets a 403 or a challenge page, and your robots.txt never comes into play.

Cloudflare offers AI crawler controls and managed robots.txt settings, and some zones have AI bot blocking switched on. Check the bot and AI crawl settings in your dashboard.
Web application firewalls may rate-limit or block unfamiliar user agents. Look for 403 responses to AI user agents in your logs.
Website builders sometimes expose an AI crawler toggle in site settings. Make sure it matches your robots.txt.
JavaScript challenges stop crawlers that don't run scripts, which includes most AI bots.

The quickest check is your server or CDN logs: filter by user agent and look at the status codes. A run of 403s for OAI-SearchBot or PerplexityBot means you're invisible to those engines whatever robots.txt says. It's one of the first things we check in an AI visibility audit.

How do you check that AI crawlers can reach your site?

Check in three places: the robots.txt file, the server logs and the rendered HTML. A crawler has to be allowed, has to get a 200 response, and has to find real content in the HTML it receives, because most AI crawlers don't run JavaScript.

Read your live robots.txt
Open yourdomain.com/robots.txt and look for groups that name AI user agents or block everything. Remove inherited rules nobody can explain.
Check your CDN and platform settings
Look for AI bot blocking, bot fight modes or JavaScript challenges, and align them with your robots.txt decision.
Search your logs by user agent
Filter for OAI-SearchBot, ChatGPT-User, PerplexityBot and Googlebot. You want 200 responses on your key pages.
Fetch a page without JavaScript
Request the page with curl or with JavaScript disabled, and confirm the headings, copy and tables are in the HTML. If not, read about server-side rendering for AI crawlers.
Watch for AI referrals
Once access is fixed, set up AI referral tracking in GA4 so you can see visits from ChatGPT and Perplexity.

Frequently asked questions

Do AI crawlers respect robots.txt?: The major AI companies, including OpenAI, Anthropic, Google and Perplexity, say their crawlers follow robots.txt, and they publish their user agents so you can write rules for them. Less reputable scrapers may ignore it or disguise themselves. robots.txt is a request, so anything truly private needs authentication or firewall rules, not a Disallow line.
Will allowing AI crawlers slow down my site?: For a typical marketing site with dozens or a few hundred pages, crawler traffic is small and static or cached pages handle it easily. Problems usually come from aggressive scrapers hitting large sites or uncached dynamic pages. If you see load spikes, rate-limit specific user agents at the CDN rather than blocking all AI crawlers in robots.txt.
Should I block AI crawlers on my staging site?: Yes, but don't rely on robots.txt alone. Put staging behind a password or an IP allowlist and add a noindex header, so neither search engines nor AI crawlers can fetch it. Then make sure the staging robots.txt and noindex settings don't get copied to production at launch, which is a classic cause of a site vanishing from search.
Can I see how often AI crawlers visit my site?: Yes. Your server, host or CDN logs record each request with its user agent, so you can count visits from GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot and others. Some CDNs, including Cloudflare, show AI crawler activity in a dashboard. GA4 won't show it, because crawlers don't run the analytics script; it only records human visits.
Does blocking AI crawlers protect my content from being copied?: Only partly. Blocking training bots stops compliant companies from collecting your pages for future training, but it doesn't remove content already collected, and it doesn't stop scrapers that ignore robots.txt. Anything public can be copied by someone. If the content is your product, put it behind a login or a paywall rather than relying on crawler rules.

Mehdi Hassan · CEO & Design Director

Copywriter, designer and developer, and founder of Designify Studio. Over five years, 60+ projects across 14+ industries, for startups that have raised $300M+.

Want this done for you? Website, $8,000.

A multi-page marketing site, designed by us, developed in Next.js and animated. $8,000, one-time, up to 6 pages. Book a free 30-minute call to see if it fits.

Schedule Call