robots.txt Generator Generate the robots.txt file for your website

robots.txt is also known as the robots exclusion standard and the robots exclusion protocol. robots.txt is a standard used by websites to communicate with web crawlers and other web robots. The standard specifies how to inform the web robot about which areas of the website should not be processed or scanned.

robots.txt is not used to "hide" data.

  1. In the top section at the left, select each of the crawlers you want to identify through robots.txt ... you can select "All" for all of the crawlers.
  2. Select your crawl delay ... keep in mind that the longer you specify, the less data crawlers will be able to find in the alloted time and span.
  3. In the Paths/Directories section, type in each directory you want to restrict access to. Each path has to start with a slash and end with a slash as in '/cgi-bin/'

If you find this tool useful, please consider a donation to keep development of free tools going. Click the link below.

Simply select a robot (or all) and enter the path or paths you wish to exclude the bot from.

To completely ban a spider from your web site, select it and enter / in one of the path/directory boxes.

To exclude different spiders from different areas, use this tool several times, once for each spider. Simply copy/paste all results, one after another, into one file.

Robots


















Crawl Delay

Sitemap (example: http://example.com/sitemap.xml)

Paths/Directories