Robots_txt 1.1

Robots_txt takes the URL of a page and retrieves the robots.txt file of the same site. The robots.txt is parsed and the rules defined in it are looked up, in order to determine if crawling a page is allowed.

Robots_txt also stores the time when a page is crawled to check whether next time another page of the same site is being crawled it is honoring the intended crawl delay and request rate limits.

Requirements:
PHP 5.0 or higher

License type: GPL - GNU General Public License
Date added: 6 years, 2 months 3 days ago | Last updated: 6 years, 2 months 3 days ago

More popular Web Search

This is actually one of the most advanced image hosting script. The imagetize key features: SEO, Ads Ready, Admin

Listing Files

robots_txt
Hot Scripts
Sponsors