Run a free scan

Blog | Answer Checkup team | Last checked

How to let AI search bots in and keep training bots out

You can stay visible in AI answers and still say no to model training. The big AI companies run different bots for each job, and robots.txt lets you treat them differently. Allow the bots that fetch pages for answers. Block the ones that collect training data.

There are three catches. Each company draws the line a little differently. Some bots that act for a user may not follow robots.txt at all. And a firewall can undo what robots.txt allows.

This post covers what each bot does and what blocking it costs you. For the full list of bots and basic copy-paste files, see our robots.txt guide for AI crawlers.

What each company's bots do

OpenAI

  • OAI-SearchBot is for search. OpenAI says sites that opt out "will not be shown in ChatGPT search answers," though they can still appear as navigational links. Keep it allowed if you want to be in ChatGPT search.
  • GPTBot is for training. Disallowing it tells OpenAI your content should not be used to train its foundation models.
  • ChatGPT-User visits a page when a user asks ChatGPT something. OpenAI says robots.txt rules may not apply to it, because a user starts the visit. It is also not used to decide whether you appear in search.

OpenAI says each setting is independent. If you allow both OAI-SearchBot and GPTBot, it may use one crawl for both jobs. Search changes take about 24 hours after you update robots.txt.

Anthropic

  • Claude-SearchBot indexes pages to improve Claude's search results. Anthropic says blocking it may reduce your visibility in those results.
  • Claude-User fetches pages when a person asks Claude a question. Blocking it stops Claude from retrieving your page for that user, which Anthropic says may reduce your visibility in user-directed search.
  • ClaudeBot collects content that could be used for training. Blocking it signals that your future content should be left out of training data.

Anthropic says its bots honor robots.txt. It also says blocking its IP addresses may not work as an opt-out, because then it can't read your robots.txt. Add the rules to robots.txt on every subdomain you want covered.

Perplexity

  • PerplexityBot surfaces and links sites in Perplexity's search results. Perplexity says it is not used to crawl content for AI foundation models.
  • Perplexity-User fetches a page when a user asks a question. Perplexity says it generally ignores robots.txt rules, since a user asked for the fetch. It is not used for training either.

Perplexity's crawler page doesn't list a training crawler. Changes can take up to 24 hours.

Google

  • Googlebot crawls for Google Search, and that includes AI Overviews and AI Mode. Google says robots.txt rules for Googlebot are the control for those features. To limit what Search shows, it points to nosnippet, data-nosnippet, max-snippet and noindex.
  • Google-Extended is not a crawler. It's a token in robots.txt. It controls whether content Google crawls may be used to train future Gemini models, and for grounding in Gemini Apps and Grounding with Google Search on Vertex AI. Google says it does not affect inclusion or ranking in Google Search.

So blocking Google-Extended keeps you out of Gemini training, but also out of that grounding. It leaves Google Search, AI Overviews included, alone. Google also says it intends to support a shared training preference signal, AIPREF, in early 2027. Watch its crawler page for that.

Example robots.txt blocks

Block training only, change nothing else

If your robots.txt already lets everyone in, add only the training groups:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

The search bots have no group of their own, so they keep following your User-agent: * group. Leave Google-Extended out if you want Gemini grounding.

The trap: a named group replaces the * group

Under the robots.txt standard, RFC 9309, a bot uses the * group only if no group matches its own name. Once you give a bot its own group, it ignores the * rules. Say your file starts like this:

User-agent: *
Disallow: /admin/
Disallow: /checkout/

If you then add User-agent: OAI-SearchBot with only Allow: /, OAI-SearchBot no longer sees the /admin/ and /checkout/ rules. Repeat them in every named group:

User-agent: OAI-SearchBot
Allow: /
Disallow: /admin/
Disallow: /checkout/

Block training on some sections only

You might want AI training to skip your long guides but not your product pages. Disallow the paths, not the whole site:

User-agent: GPTBot
Disallow: /guides/
Disallow: /research/

User-agent: ClaudeBot
Disallow: /guides/
Disallow: /research/

Search bots still read everything.

What robots.txt can't do

  • User-triggered fetchers. OpenAI says robots.txt may not apply to ChatGPT-User. Perplexity says Perplexity-User generally ignores it. Anthropic says Claude-User follows it. If you need a hard block, that's access control on your server, not robots.txt. The robots.txt standard itself says its rules are not a form of access authorization.
  • Firewalls and CDNs. A bot setting can block OAI-SearchBot or PerplexityBot even when robots.txt allows them. Perplexity's crawler page has steps for Cloudflare and AWS WAF that match the user agent together with its published IP ranges. Our curl test with AI user agents shows what your server sends each bot, and its limits.
  • Fake bots. Anyone can send a bot's name as the user agent. OpenAI, Perplexity and Anthropic publish their IP ranges, so you can check a request is real before you trust it or block it.

Check that it worked

  1. Fetch https://yoursite.com/robots.txt and read every group. Look for old groups from a plugin.
  2. Wait at least 24 hours. OpenAI and Perplexity both give that as their update time.
  3. Search your server or CDN logs for OAI-SearchBot, PerplexityBot and Claude-SearchBot. Check the IPs against the published lists.
  4. Run our free scan. Its score includes a robots.txt check that weighs search bots more than training bots: 6 points each for OAI-SearchBot, ChatGPT-User and PerplexityBot, 3 points each for GPTBot, ClaudeBot, Google-Extended and CCBot. It looks at access to your homepage. Blocking training bots is a fair choice, and the scan treats it as low priority.

Access is the first step, not the last. Being cited also takes pages worth quoting and coverage on other sites. Our AI SEO guide covers that part.

FAQ

Will blocking GPTBot remove me from ChatGPT answers?
No. OpenAI keeps the two apart. GPTBot is for training and OAI-SearchBot is for search, and each setting is independent. Block GPTBot, keep OAI-SearchBot, and you stay eligible for ChatGPT search.
Does blocking Google-Extended affect AI Overviews?
No. Google says Google-Extended doesn't affect Google Search. AI Overviews and AI Mode follow your Googlebot rules and snippet controls, so you keep your Search reach.
Why do I still see ChatGPT-User or Perplexity-User after blocking them?
Both act for a person who asked a question. OpenAI says robots.txt may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores it. The upside: those visits come from real questions, and Perplexity says it may include a link to your page in its answer.
Should I block Claude-SearchBot to keep my content out of training?
No need. ClaudeBot is Anthropic's training bot. Block ClaudeBot and keep Claude-SearchBot, so you stay visible in Claude's search results.

Run the free scan

Run the free scan on the Answer Checkup homepage. It runs all six checks on any page, including which AI crawlers your robots.txt lets in, and gives you a score and a grade. You can unlock the full checks and fixes. See pricing for what the paid report adds.

Answer Checkup is not affiliated with OpenAI, Perplexity, Google, Anthropic or Common Crawl.

Sources

All pages fetched and checked on November 3, 2026.