Robots.txt generator, with AI crawlers built in
Set blocked paths and your sitemap, then choose which AI assistants may find, cite or train on your pages. We write the robots.txt.
Search engines and AI assistants
Who may train AI models on your pages
Blocked paths and sitemap
robots.txt
User-agent: *
Allow: /
Put it at the root of your site, like yourwebsite.com/robots.txt. If you already have one, merge these groups into it rather than replacing rules you need. Then check it with the AI crawler checker.
Make a robots.txt for Google, Bing and AI crawlers
A robots.txt file tells crawlers which parts of your site they may read. This free robots.txt generator writes the usual rules for search engines, plus choices for AI search and model training. Everything happens in your browser. Copy or download the file when you're done.
For a basic Google robots.txt, leave Allow everything selected. Add any paths you want crawlers to skip, such as /admin/ or /cart/, one per line under Keep every crawler out of. Add your sitemap's full URL if you have one. Leave the paths empty to allow the whole site. The AI switches are optional.
With no blocked crawlers or paths, the file starts with:
User-agent: *
Allow: /
User-agent: * covers crawlers without a group of their own. Each blocked path gets a Disallow line. The sitemap gets a Sitemap line, which points crawlers to your list of pages; it doesn't guarantee those pages will be indexed.
robots.txt controls crawling, not access. Protect private pages with a login. If you need to keep a public page out of Google Search, use noindex and let Google crawl the page so it can read that instruction. A blocked URL can still appear in search results without its content being crawled.
Search crawlers and training crawlers are separate choices
Most AI companies run one crawler to find pages their assistant can cite, and another to collect pages for training models. robots.txt names them separately, so you can allow search while blocking training:
| Company | Finds pages to cite | Trains models |
|---|---|---|
| OpenAI | OAI-SearchBot, ChatGPT-User | GPTBot |
| Googlebot | Google-Extended | |
| Anthropic | Claude-SearchBot, Claude-User | ClaudeBot |
| Perplexity | PerplexityBot, Perplexity-User | None listed |
| Microsoft | Bingbot | None separate |
Blocking a training crawler doesn't take you out of that company's search answers, with one exception: Google-Extended also covers grounding in some of Google's AI products, like the Gemini app, though not Google Search. Blocking a search crawler is different: OpenAI says sites that block OAI-SearchBot won't be shown in ChatGPT search answers, and Anthropic says blocking Claude-SearchBot may make you less visible in Claude's.
Two crawlers that are also search engines
Googlebot feeds Google Search as well as AI Overviews and AI Mode, and Bingbot feeds Bing as well as Copilot. robots.txt can't keep their AI out while leaving search alone. Blocking them stops them reading your pages, which hurts you in ordinary search too: a blocked page can still be listed if other sites link to it, but without a description. Google and Bing offer page-level controls instead: nosnippet for Google, and noarchive or nocache for Copilot.
Fetches made for a person
ChatGPT-User, Claude-User and Perplexity-User open a page when someone asks an assistant about it. Anthropic says Claude-User follows robots.txt. OpenAI and Perplexity say theirs may not, because a person asked, so blocking them is a request rather than a guarantee.
How the generated groups work
Crawlers you switch off share one group that disallows everything. Every other crawler falls under User-agent: *, which only keeps it out of the paths you list. A crawler follows the group that names it and ignores *, so the two never mix. The Allow everything preset resets the crawler switches; it leaves your blocked paths and sitemap in place. Clear the paths too if you want no restrictions.
How to install and test your robots.txt
- Copy or download the file. If your site already has a robots.txt, merge the rules you need before replacing it. Check for existing crawler groups that could change which rules apply.
- Publish it as plain text at your site's root, such as
https://example.com/robots.txt. It won't work from/blog/robots.txt. Each subdomain needs its own file. Website builders may have a setting for this instead of a file upload. - Open that URL in a browser and check that it shows the text, not an error or a page from your website. Google's installation guide covers the file requirements.
- In Search Console, open the robots.txt report. Check the version Google fetched, its last crawl time, and any warnings or errors. Use a Domain property or a URL-prefix property at the site root. You can request a recrawl after fixing a critical error; changes aren't instant.
To check what your live file allows for AI crawlers, run the AI crawler checker. It also tests whether your firewall turns those crawlers away.
Robots.txt generator questions
Both. The default group works for Googlebot, Bingbot and other crawlers that follow robots.txt. You can set blocked paths and a sitemap without changing any AI switches.
Choose Allow everything and clear Keep every crawler out of. The output will allow the whole site. You can keep the sitemap line.
No. Anyone can read the file and visit the paths it lists. Use authentication for private content. A Disallow rule asks crawlers to stay out; it doesn't protect the page or reliably remove its URL from search results.
Choose Search yes, training no. It blocks the listed training crawlers while leaving search crawlers allowed. Google-Extended also controls some Gemini grounding, and fetches made for a person may follow different rules, as explained above.
Find out what AI says
about your business.
Grow AI visibility of your business and bring new customers.