Free tool
AI crawlers: what do you let through?
AI companies send robots to read the web: some collect text to train their models, others look up pages to answer their users. We read your robots.txt and tell you, robot by robot, what it allows. Then you choose.
Generator: your choice, as robots.txt
Tick what you want to block. The block updates as you go; paste it at the end of your robots.txt. Everything happens in your browser.
Nothing else to change: Googlebot and Bingbot are untouched, your search ranking stays the same. On WordPress, an SEO plugin (Yoast, Rank Math) lets you edit robots.txt without file access.
Three kinds of AI robots
Blocking “AI” as a whole makes little sense: each company runs several robots, for different purposes. There is no right answer; it depends on what you publish and what you expect from it.
01
Training
GPTBot, ClaudeBot, Google-Extended, CCBot… They gather text to train future models. Blocking them keeps your content out of the next models, with no effect on your Google ranking.
02
AI search
OAI-SearchBot, Claude-SearchBot, PerplexityBot… They build the index an AI tool uses to answer and cite its sources. Blocking them can remove your site from those answers.
03
On demand
ChatGPT-User, Claude-User, Perplexity-User… They visit a page because a user asked. It's close to a human visit.
Good to know
Google-Extended and Applebot-Extended are not robots that visit: they are names read by Googlebot and Applebot to know whether your pages may be used for their AI. Blocking them doesn't change your presence in Google Search. Google's AI Overviews, however, depend on Googlebot itself.
robots.txt is a courtesy: well-known companies respect it, less scrupulous robots ignore it. To truly block a robot, you need a firewall rule (Cloudflare, for example, offers one-click AI blocking).
Frequently asked questions
Which AI crawlers does my site let through?
The tool reads your robots.txt and tells you, robot by robot, what it allows: GPTBot, ClaudeBot, Google-Extended, PerplexityBot and the others. A generator then helps you choose what to permit.
Should I block every AI crawler?
There is no right answer. Training bots feed future models, search bots feed the answers AI tools give, and on-demand bots visit because a user asked. Blocking search bots can remove your site from those answers.
Does blocking Google-Extended hurt my search ranking?
No. Google-Extended and Applebot-Extended are names that Googlebot and Applebot read to know whether your pages may be used for their AI. Google's AI Overviews, however, depend on Googlebot itself.
Does robots.txt really stop the bots?
It is a courtesy: well-known companies respect it, less scrupulous robots ignore it. To truly block a robot, you need a firewall rule at your host or CDN.
