Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.

About /robots.txt In a nutshell. Web site owners use the /robots.txt file to give instructions about their site to web robots; this is called The Robots Exclusion ...

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web ...

This example "/robots.txt" file specifies that no robots should visit any URL starting with "/cyberworld/map/", except the robot called "cybermapper":

User-agent: * Disallow: /search Disallow: /sdch Disallow: /groups Disallow: /images Disallow: /catalogs Allow: /catalogs/about Allow: /catalogs/p? Disallow: /catalogues

Information on the robots.txt and how it effects your website. Also includes a free robots.txt generator

robots.txt generator designed by an SEO for public use. Includes tutorial.

Learn about the robots.txt, and how it can be used to control how search engines and crawlers do on your site.

Use this module when you are running multiple Drupal sites from a single code base (multisite) and you need a different robots.txt file for each one.

# robots.txt for http://www.wikipedia.org/ and friends # # Please note: There are a lot of pages on this site, and there are # some misbehaved spiders out there that ...