Robots.txt Generator
Generate a robots.txt file for your website in seconds. Control which search engines and AI crawlers can access your content — then download and upload to your site root.
Default rule
Disallow paths
/. Example: /private/Bot rules
How to use this generator
Fill in the settings on the left and the robots.txt file updates in real time. When you're happy with the output:
- Click Download robots.txt to save the file.
- Upload it to the root directory of your website — e.g.
https://example.com/robots.txt - That's it. Search engines check this location automatically on every crawl.
What is a robots.txt file?
A robots.txt file is a plain text file placed at the root of your website that tells web crawlers which pages or sections they are allowed or not allowed to crawl. It follows the Robots Exclusion Protocol (REP) — an informal standard respected by all major search engines.
A basic robots.txt file looks like this:
User-agent: * Disallow: /private/ Allow: / Sitemap: https://example.com/sitemap.xml
User-agent: * means "all crawlers". Disallow: /private/ tells them not to crawl anything inside that folder. Allow: / explicitly permits everything else.
AI crawlers in 2026 — what's new
In 2026, a growing number of AI companies crawl the web to train large language models. Unlike traditional search bots, these crawlers don't send traffic back to your site. Many site owners now block them to protect their content from being used as training data without permission.
Key AI crawlers you may want to control:
- GPTBot — OpenAI's training crawler. Block with
User-agent: GPTBot/Disallow: / - ClaudeBot — Anthropic's crawler used for Claude training data.
- PerplexityBot — Perplexity AI's crawler for its answer engine.
- Google-Extended — Google's opt-out token for Gemini and Vertex AI training (separate from Googlebot).
- CCBot — Common Crawl, used by many AI companies as a training data source.
- Bytespider — ByteDance (TikTok parent) crawler, increasingly used for AI training.
Blocking these bots does not affect your Google or Bing search rankings — they use separate bots (Googlebot, Bingbot).
Robots.txt directives explained
User-agent— Specifies the crawler the rule applies to. Use*for all bots.Disallow— Tells the bot not to crawl a specific path.Disallow: /blocks the entire site.Allow— Explicitly permits a path, even if a parent directory is disallowed.Crawl-delay— Requests the bot to wait N seconds between requests. Not supported by Googlebot.Sitemap— Points crawlers to your XML sitemap. Can appear anywhere in the file.
Frequently asked questions
https://yourdomain.com/robots.txt. It must be accessible without authentication. Placing it in a subfolder (e.g. /blog/robots.txt) does nothing — crawlers only check the root.noindex meta tag or X-Robots-Tag HTTP header instead.User-agent blocks as you need, each with their own Allow/Disallow rules. The generator above handles this automatically when you set individual bot rules.yourdomain.com/robots.txt directly in a browser to confirm the file is live.