robots.spider(tm) 1.0

robots.spider(tm) will read a robots.txt file and compare it with the pages on the site. Search engines spider the site. If there are no references to a directory or file, it will not be indexed.

robots.txt files should not refer to unreferenced content because it allows other viewers to identify areas of your site that you don't want indexed. The read depth limits the number of pages scanned.

License type: GPL
Date added: 10 years, 1 month 16 days ago | Last updated: 10 years, 1 month 16 days ago

More popular Configuration

This is actually one of the most advanced image hosting script. The imagetize key features: SEO, Ads Ready, Admin

Listing Files

Hot Scripts