Can AI assistants read your site?

What this checks

AI assistants find pages to cite with their own crawlers. If your site turns those crawlers away, the assistant has less of yours to work with. This checker reads your site the way they would:

• robots.txt: whether each AI crawler may read your homepage, using the same rules the crawlers follow. We fetch robots.txt and your homepage as an ordinary browser, so a site that answers crawlers differently may show different results.
• Your firewall: we also ask for your homepage using the names of OAI-SearchBot, Claude-SearchBot and PerplexityBot. Some bot protection turns them away whatever robots.txt says. The real crawlers come from their companies' own addresses, and firewalls often block requests that only claim to be a crawler, as ours do. So a block here is a sign to check your settings rather than proof.
• Text without JavaScript: most AI crawlers don't run JavaScript. If your words only appear after scripts run, they may see an empty page.
• noindex and nosnippet: Google's AI Overviews and AI Mode only cite pages that can appear in Google Search with a snippet.
• Business details: whether your homepage states who you are in structured data. It's optional; the local business schema generator writes it.
• Sitemap and llms.txt: a sitemap helps crawlers find your pages. llms.txt is optional; no major assistant has said it uses it.

Each assistant also has its own page that puts its result first and explains how it finds sites: ChatGPT, Google's AI, Claude, Perplexity and Copilot.

Search crawlers and training crawlers

Each AI company runs separate crawlers for different jobs, and robots.txt treats them separately.

• Search and answer crawlers (OAI-SearchBot, Googlebot, Bingbot, Claude-SearchBot, PerplexityBot) collect the pages assistants cite when they answer. Block these and you're likely to be left out of their answers.
• Fetches made for a person (ChatGPT-User, Claude-User, Perplexity-User) happen when someone asks an assistant about a page. OpenAI and Perplexity say theirs may not follow robots.txt, because a person asked; Anthropic says Claude-User does.
• Training crawlers (GPTBot, ClaudeBot, CCBot) and Google-Extended decide whether your pages may be used to train models. Blocking them doesn't remove you from ChatGPT search or Google Search, so it's a separate choice. Google-Extended also covers grounding in some of Google's AI products.

How to let a crawler in

robots.txt lives at the root of your site, like yourbusiness.com/robots.txt. A file that lets every crawler read everything looks like this:

User-agent: *
Allow: /

Sitemap: https://yourbusiness.com/sitemap.xml

To allow the search crawlers but keep your pages out of model training, name the training crawlers separately:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Google-Extended
Disallow: /

User-agent: *
Allow: /

The robots.txt generator writes this file for you. OpenAI says changes take about a day to reach ChatGPT search. If robots.txt allows a crawler but this checker shows your server turning it away, look at your host's or CDN's bot settings: some block AI crawlers by default.

What this doesn't tell you

Being readable is the first step, not the result. It doesn't tell you whether ChatGPT or Google's AI Mode actually names you when customers ask. For that, see how to get ChatGPT to recommend your business, work through the AI visibility checklist, or let Glowsearch ask your customers' questions every week and show you who gets named instead.