WerklyTools
Esc
↑↓ navigate↵ open
EN
Buy me a coffee

Robots.txt generator

The robots.txt file tells Google and other crawlers which parts of your site they may visit. Pick a template, tweak the rules per crawler, add your sitemap and download the file. The tool warns you about common mistakes, such as blocking your whole site by accident.

Processed 100% in your browser

Templates

robots.txt is a voluntary request: well-behaved crawlers follow it, but it does not protect private information. Upload it to the root of your domain (yoursite.com/robots.txt).

robots.txt

Ctrl + Enter runCtrl + C copy result

How to use

  1. 1Pick a template: allow all, block all, WordPress or block AI.
  2. 2Adjust the allowed and blocked paths for each crawler.
  3. 3Add the full address of your sitemap.
  4. 4Download the file and upload it to the root of your domain.

Use cases

  • Keep the admin area or internal search pages out of the index.
  • Stop AI crawlers from using your content to train models.
  • Declare the sitemap so search engines discover your pages.
  • Review an old robots.txt before launching a new site.

Frequently asked questions

Where do I upload robots.txt?

To the root of the domain, so it opens at yoursite.com/robots.txt. A robots.txt inside a folder is not valid.

Does robots.txt hide pages from Google?

Not completely: it asks crawlers not to visit them, but a blocked page can still appear if others link to it. To exclude it, use the noindex tag.

Can it protect private information?

No. It is a voluntary request and anyone can read it. Protect sensitive data with a password, not robots.txt.

Which AI crawlers can I block?

The template includes the best-known ones, such as GPTBot, ClaudeBot, CCBot and Google-Extended. Only those that respect the standard will comply.

Related tools

Base64 encoder and decoderConvert text or any file to Base64 and back. Supports the URL-safe variant, line breaks and data URI prefixes, and detects when decoded content is binary.
Code to imageCreate attractive code screenshots for social media, slides, blogs or documentation. Paste your code, pick a language, theme and background, and download a crisp PNG or a scalable SVG.
Cron expression generator and translatorType a cron expression or build it field by field, and the tool tells you in plain words when it will run, plus the next ten dates. That way you can confirm a scheduled task will do what you expect before putting it on the server.
CSS minifier and beautifierShrink your stylesheets by removing comments, whitespace and empty rules, or do the opposite: turn minified CSS into readable, indented code. You will see the original size, the final size and the gzipped size.
HTML formatter and minifierPaste messy or single-line HTML and get clean, indented code that is easy to read and review. You can also do the opposite and minify it to make it smaller. Everything happens in your browser and your code is never sent anywhere.
JavaScript minifier and beautifierCompress your JS by stripping comments and whitespace and shortening names, or format a minified script so it is readable. Powered by Terser, right in your browser.