Free tool
AI Crawler robots.txt Generator
Decide which AI crawlers can read your site. Tick a bot to block it, then copy the rules into your robots.txt file. Every bot you leave unticked gets an Allow rule.
Free. Runs in your browser. Nothing you type is sent or stored.
Choose the bots to block
Updates as you tickBefore you block: blocking a search or answer crawler, such as OAI-SearchBot, Claude-SearchBot or PerplexityBot, can remove your pages from that tool's answers and citations.
Google-Extended only controls whether Google may use your content for Gemini training. It is separate from Googlebot, so blocking it does not change your Google Search listings.
How to use it
Set your AI crawler rules
Pick a starting point
Use a preset, or tick each bot you want to block.
Add your sitemap
Optional. Paste your full sitemap URL to add a Sitemap line.
Copy the rules
Press Copy to put the rules on your clipboard.
Update robots.txt
Add the rules to the robots.txt at your domain root, next to your current rules.
FAQ
Questions about AI crawlers and robots.txt
What happens if I block an AI search crawler?
Blocking a search or answer crawler such as OAI-SearchBot, Claude-SearchBot or PerplexityBot can stop that tool from finding your pages, so they may drop out of its answers and citations. If you want AI tools to cite you, leave these allowed.
Does blocking Google-Extended remove my site from Google Search?
No. Google-Extended is a control token, not a crawler. Googlebot keeps crawling your site for Search as usual, and the token only tells Google whether that content may also be used for Gemini training.
Can I block AI training but stay visible in AI search?
Often, yes. OpenAI and Anthropic, for example, run separate bots for training and for search. The preset on this page blocks the training bots and allows the search and user bots. Check each company's documentation, because bot names and roles change.
Do all AI crawlers follow robots.txt?
robots.txt works on trust. The large AI companies say their named crawlers follow it, though user-triggered visits can be handled differently, and some crawlers ignore it altogether. If you need a hard block, use firewall or server rules.
Where do I add these AI crawler rules?
Add them to the robots.txt file at the root of your domain, alongside your existing rules. Do not delete your current rules for Googlebot or other search engines unless you mean to.
Check what AI tools say about you
These files make your site easier for AI tools to read. To find out whether ChatGPT, Perplexity and Google AI Overviews actually name your business, ask for a free check.