🌓
robots.txt Generator
Control how search engines crawl your site
Quick Presets
Allow All
Block All
Standard
Block AI Bots
Block Bad Bots
User-Agent
* (All bots)
Googlebot
Bingbot
GPTBot (OpenAI)
ChatGPT-User
Claude-Web
Bytespider (TikTok)
CCBot (Common Crawl)
Twitterbot
Facebook
Sitemap URL (optional)
Crawl-Delay (optional, seconds)
Rules
Disallow
Allow
Add
Common paths:
/admin/
/api/
/private/
/tmp/
/*.pdf$
robots.txt Output
📋 Copy
⬇ Download