Robots.txt Overview
Web-Indexing Robots can be controlled by installing a file in the root path of the server with explicit instructions on where a spider is not permitted. This method relies on Robots / Spiders obeying the robots.txt file, and does not always happen with every Robot. If you want additional protection from robots/spider, you may wish to use methods such as password protecting the directory you wish to keep private You can define which paths are off limits for spiders to visit. This is useful for such things as large directories of information, personal information, and parts of the website containing large amounts of recursive links, among others. Enter each disallowed path, one per line, in the box below. Each path must end with a "/" Robots.txt examples: To allow all robots/spiders index your website User-agent: * Disallow: ...