bw
better
-web
.eu
Home
Features
Tools
About
Contact
EN
Deutsch
English
robots.txt Generator
Creates a robots.txt with targeted AI bot blocking for GPTBot, ClaudeBot and 20+ more crawlers.
Block AI crawlers
GPTBot
(OpenAI)
ClaudeBot
(Anthropic)
anthropic-ai
(Anthropic)
CCBot
(Common Crawl)
PerplexityBot
(Perplexity AI)
Google-Extended
(Google)
Bytespider
(ByteDance)
Amazonbot
(Amazon)
FacebookBot
(Meta)
Applebot-Extended
(Apple)
Diffbot
(Diffbot)
cohere-ai
(Cohere)
ImagesiftBot
(Imagesift)
omgili
(Webz.io)
Timpibot
(Timpi)
YouBot
(You.com)
PetalBot
(Huawei)
Webzio-Extended
(Webz.io)
DataForSeoBot
(DataForSEO)
TurnitinBot
(Turnitin)
AI2Bot
(Allen AI)
Scrapy
(Scrapy)
Sitemap URL (optional)
Crawl delay in seconds (optional)
Additional disallow paths (optional, one per line)