Blog · Answer Checkup team · Last checked

Can AI crawlers see my site? Six checks to run

To find out whether AI crawlers can see your site, check six things: robots.txt, your main content in the raw HTML, structured data, title and description, llms.txt, and your sitemap. Each takes a minute or two with a terminal.

These are the six checks our free scan runs, with the same points.

The six checks and their points

CheckPoints
AI crawler access in robots.txt30
Content present in the raw HTML20
Structured data (JSON-LD)15
Title and meta description15
llms.txt10
Sitemap10
Total100

robots.txt, llms.txt and the sitemap are read from your site's root, the rest from the page you scan. 90 or more is an A.

1. AI crawler access in robots.txt (30 points)

The scan reads robots.txt and works out whether each of seven AI crawlers may fetch your homepage. Search and user-fetch bots count for more than training bots:

  • 6 points each: OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot (Perplexity).
  • 3 points each: GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google), CCBot (Common Crawl).

A bot that's blocked from the homepage gets 0. A bot that's allowed on the homepage but blocked from a few paths, like /admin, still gets full points, because that's normal.

Blocking training bots is a fair choice, so the scan lists it as low priority. OpenAI's docs say each setting is independent: you can allow OAI-SearchBot for ChatGPT search and disallow GPTBot for training. They also say robots.txt rules may not apply to ChatGPT-User, because a user starts those visits.

Check it by hand:

curl -sL -o /dev/null -w "%{http_code} %{content_type}\n" https://example.com/robots.txt
curl -sL https://example.com/robots.txt

You want a 200 and plain text. If you get a 301 without -L, your site redirects (for example to www). That's fine; our scan follows up to three redirects. Then read every group: User-agent: * with Disallow: / blocks every bot without its own group.

A firewall or CDN bot setting can also block bots. A rough test is to send the bot's name as the user agent:

curl -sL -o /dev/null -w "%{http_code}\n" -A "OAI-SearchBot" https://example.com/

A 403 suggests a user-agent rule. A 200 doesn't prove the real bot gets through, since some firewalls check IP addresses too.

Fix it: remove Disallow: / from the groups for the search bots, or add an explicit Allow: / group for each. Our robots.txt guide for AI crawlers has copy-paste rules, including one that allows AI search and blocks training.

2. Content present in the raw HTML (20 points)

Many crawlers read the HTML your server sends without running JavaScript. The scan strips scripts, styles and tags and counts the words left. 150 words or more earns 15 points. 50 to 149 words earns 7. One <h1> with text earns the other 5.

If the page has fewer than 50 words and an empty app container or external scripts, the scan flags it as likely client-rendered.

Check it by hand:

curl -sL https://example.com/ | perl -0pe 's/<script\b.*?<\/script>//gis; s/<style\b.*?<\/style>//gis; s/<[^>]+>/ /g' | wc -w

It's a rough count. If it's near zero on a page full of text in your browser, crawlers that skip JavaScript see an empty page.

Fix it: render your main content on the server or at build time, so what you do, who it's for and where you are is in the HTML. Our post on JavaScript and AI crawlers covers how to test this properly and the fixes.

3. Structured data (15 points)

The scan looks for schema.org JSON-LD blocks in the page. At least one block that parses as valid JSON earns 8 points. A type that describes your business, such as Organization, LocalBusiness, Product, Service or WebSite, earns the other 7.

Check it by hand:

curl -sL https://example.com/ | grep -c 'application/ld+json'

A 0 means no JSON-LD in the raw HTML. If it's there, check that the block parses as JSON.

Fix it: add an Organization or LocalBusiness block to your homepage, plus Product, Service or Offer for what you sell. Only describe things that are visible on the page.

4. Title and meta description (15 points)

The title earns 7.5 points if it's 10 to 70 characters, and the description earns 7.5 if it's 50 to 170. Each one that exists but falls outside that range gets 4.

Check it by hand:

curl -sL https://example.com/ | grep -io '<title[^>]*>[^<]*</title>'
curl -sL https://example.com/ | grep -io '<meta[^>]*name=.description.[^>]*>'

Fix it: a title that says what you do and for whom, and a description with your offer, audience or location, and one concrete fact. Both should be in the HTML your server sends, not set later by a script.

5. llms.txt (10 points)

llms.txt is an optional markdown file that summarizes your site for AI agents. The scan gives 5 points if /llms.txt returns a 200 and isn't an HTML page (some sites serve their homepage for missing files), and 5 more if the first line is an H1 with your name and the file has at least one markdown link.

Check it by hand:

curl -sL https://example.com/llms.txt | head -5

Fix it: use the template in our llms.txt guide. No engine has said it uses llms.txt to choose citations, so treat it as low stakes.

6. Sitemap (10 points)

The scan looks at any Sitemap: line in robots.txt, then /sitemap.xml and /sitemap_index.xml, counting only sitemaps on your own site. Finding one earns all 10 points. If robots.txt doesn't point to it, the scan adds a low-priority fix.

Check it by hand:

curl -sL https://example.com/robots.txt | grep -i '^sitemap:'
curl -sL -o /dev/null -w "%{http_code}\n" https://example.com/sitemap.xml

Fix it: publish a sitemap and add its full URL to robots.txt. The sitemaps.org protocol says this line is independent of the user-agent lines, so it can go anywhere:

Sitemap: https://example.com/sitemap.xml

Our own site, scanned with our own tool

On October 15, 2026, we scanned our own new preview site with our own tool, and it scored 100 out of 100, grade A, on the homepage and on two blog posts. All seven crawlers were allowed, llms.txt was valid, JSON-LD included Organization, and the homepage had 713 words in its raw HTML.

The scan still flagged one fix, on all three pages: our robots.txt had only these two lines and no Sitemap: line.

User-agent: *
Allow: /

The sitemap was at the default address, so the score didn't drop. The fix is the one Sitemap: line from check 6.

A perfect score doesn't mean AI answers mention you

The six checks tell you whether AI crawlers can read your site. They don't tell you whether AI answers name you.

We measure that separately with our answer check, and we haven't run it on the new preview site yet. We did run it on October 15, 2026 on our previous site, showsupin.com, a one-page "launching soon" placeholder that was one day old: 25 buyer questions, 3 times on each engine, through the OpenAI API with web search and the Perplexity API, for 150 answers. It scored 0 out of 100. None of the 150 answers mentioned it or cited it. The answers made 510 citations, and 491 of them went to sites other than ours or the three competitors we tracked, mostly review sites, articles and comparison posts.

So crawler access is the first step, not the last. Our AI SEO guide covers those steps, and how to check if ChatGPT recommends your business shows how to measure where you stand.

FAQ

How do I check if GPTBot can crawl my site?
Open https://yoursite.com/robots.txt and look for a User-agent: GPTBot group, or a User-agent: * group if GPTBot has none. If the matching group has Disallow: /, GPTBot is blocked. Remember that GPTBot is OpenAI's training crawler. OAI-SearchBot is the one for ChatGPT search.
If robots.txt allows everything, can AI see my site?
Not always. A firewall can still block bots, and a page that only renders in the browser can look empty to crawlers.
Does a 100 score mean ChatGPT will recommend me?
No. It means crawlers can read your site. Our previous site, a one-day-old placeholder, scored 0 out of 100 on our answer check, and almost all the citations in those answers went to other sites. Being cited takes coverage elsewhere, not just a readable site.

Run the free scan

Run the free scan on the Answer Checkup homepage. It runs all six checks on any page and lists what to fix first. For the research behind visibility, see our guide to generative engine optimization.

Answer Checkup is not affiliated with OpenAI, Perplexity, Google, Anthropic or Common Crawl.
Contact: [email protected]

Sources

All pages fetched and checked on October 15, 2026.