Skip to content
bw
better
-web
.eu
Home
Features
Domain management
Tools
About
Contact
Log in
Register
System default
Light
Dark
EN
Deutsch
English
Home
Features
Domain management
Tools
About
Contact
Log in
Register
robots.txt Generator
Creates a robots.txt with targeted AI bot blocking for GPTBot, ClaudeBot and 20+ more crawlers.
Block AI crawlers
GPTBot
(OpenAI)
ClaudeBot
(Anthropic)
anthropic-ai
(Anthropic)
CCBot
(Common Crawl)
PerplexityBot
(Perplexity AI)
Google-Extended
(Google)
Bytespider
(ByteDance)
Amazonbot
(Amazon)
FacebookBot
(Meta)
Applebot-Extended
(Apple)
Diffbot
(Diffbot)
cohere-ai
(Cohere)
ImagesiftBot
(Imagesift)
omgili
(Webz.io)
Timpibot
(Timpi)
YouBot
(You.com)
PetalBot
(Huawei)
Webzio-Extended
(Webz.io)
DataForSeoBot
(DataForSEO)
TurnitinBot
(Turnitin)
AI2Bot
(Allen AI)
Scrapy
(Scrapy)
Sitemap URL (optional)
Crawl delay in seconds (optional)
Additional disallow paths (optional, one per line)