Can Claude see your website?

How Claude finds websites

When Claude searches the web, two things decide what it can find: Anthropic's own crawlers, and the search index it queries.

Anthropic runs three crawlers, and says they all follow robots.txt:

Unlike OpenAI and Perplexity, Anthropic says its user-triggered fetcher, Claude-User, follows robots.txt too. Block it and Claude won't fetch your page even when someone asks it about you.

The search engine behind Claude

Anthropic hasn't said publicly which search index Claude queries. But in March 2025, when Claude got web search, Anthropic added Brave Search to its list of subprocessors, and Simon Willison found that Claude's citations matched Brave's top results for the same query.

That matters because Brave's crawler works differently. Brave says its crawler doesn't announce a name of its own, so sites can't single it out, and that it won't crawl a page Googlebot isn't allowed to. So:

• Blocking Googlebot hides you from Brave too, and so probably from Claude's search, whatever your rules for Anthropic's crawlers say.
• You can't allow or block Brave by name. Your Googlebot rules decide it.
• To remove a page from Brave, Brave says to use noindex rather than robots.txt, and lets you ask for a page to be fetched again so a change shows up sooner.

Brave's index is smaller than Google's. A page Google knows about can be missing from Brave, and new sites can take a while to appear. Searching your own site on search.brave.com is a quick way to see what Claude's search may have to work with.

What this checks

• Whether your robots.txt lets Claude-SearchBot, Claude-User and ClaudeBot read your homepage, and which line decides it.
• Whether your server lets in a request that uses Claude-SearchBot's name. Bot protection sometimes turns AI crawlers away whatever robots.txt says. The real crawler comes from Anthropic's published addresses, so treat a block here as a sign to check your settings.
• Whether your homepage has its text in the HTML. In Vercel's December 2024 measurements, Anthropic's crawler downloaded JavaScript files but didn't run them.

If Claude is blocked

Allow Claude-SearchBot and Claude-User. To stay out of training, block ClaudeBot on its own:

User-agent: Claude-SearchBot
User-agent: Claude-User
Allow: /

User-agent: ClaudeBot
Disallow: /

The robots.txt generator writes this for you. A few things specific to Anthropic:

• Each subdomain needs its own file. Anthropic reads robots.txt per subdomain, so a shop or blog on its own subdomain follows its own rules.
• Slow it down instead of blocking it. Anthropic honours Crawl-delay, for example Crawl-delay: 1 under its user agent.
• Don't block it by address. Anthropic warns that blocking its IP addresses may not work reliably, because then it can't read your robots.txt. Use robots.txt to opt out.
• CAPTCHAs stop it. Anthropic says its crawlers won't try to get past CAPTCHAs or similar protection. If your host puts a challenge page in front of visitors, Claude sees the challenge, not your page.

How to tell whether Claude is reading you

Your server logs. Search for Claude-SearchBot, Claude-User and ClaudeBot in the user agent. Anthropic publishes its crawlers' addresses at claude.com/crawling/bots.json; a request from one of them really is Anthropic.

Claude itself. With web search on, ask Claude a question your page answers, then look at the sources. Ask more than once, since answers vary.

Brave Search. Search for your business and your main pages on Brave. If they're missing there, they're likely missing from Claude's search too.

If Anthropic's crawlers misbehave on your site, Anthropic asks you to email claudebot@anthropic.com from an address at the domain concerned.

Readable isn't the same as recommended

Being readable lets Claude cite you; whether it does depends on what it finds about you. Why AI names a competitor instead covers the usual reasons, and on the Pro plan, Glowsearch asks Claude your customers' questions every week and shows you who it names.