Free tool
AI crawlers: what do you let through?
AI companies send robots to read the web: some collect text to train their models, others look up pages to answer their users. We read your robots.txt and tell you, robot by robot, what it allows. Then you choose.
Generator: your choice, as robots.txt
Tick what you want to block. The block updates as you go; paste it at the end of your robots.txt. Everything happens in your browser.
Nothing else to change: Googlebot and Bingbot are untouched, your search ranking stays the same. On WordPress, an SEO plugin (Yoast, Rank Math) lets you edit robots.txt without file access.
Three kinds of AI robots
Blocking “AI” as a whole makes little sense: each company runs several robots, for different purposes. There is no right answer; it depends on what you publish and what you expect from it.
01
Training
GPTBot, ClaudeBot, Google-Extended, CCBot… They gather text to train future models. Blocking them keeps your content out of the next models, with no effect on your Google ranking.
02
AI search
OAI-SearchBot, Claude-SearchBot, PerplexityBot… They build the index an AI tool uses to answer and cite its sources. Blocking them can remove your site from those answers.
03
On demand
ChatGPT-User, Claude-User, Perplexity-User… They visit a page because a user asked. It's close to a human visit.
Good to know
Google-Extended and Applebot-Extended are not robots that visit: they are names read by Googlebot and Applebot to know whether your pages may be used for their AI. Blocking them doesn't change your presence in Google Search. Google's AI Overviews, however, depend on Googlebot itself.
robots.txt is a courtesy: well-known companies respect it, less scrupulous robots ignore it. To truly block a robot, you need a firewall rule (Cloudflare, for example, offers one-click AI blocking).
