Robots_txt 1.1
Developed by Andy Pieters, listed in Web Search
Robots_txt takes the URL of a page and retrieves the robots.txt file of the same site. The robots.txt is parsed and the rules defined in it are looked up, in order to determine if crawling a page is allowed.
Robots_txt also stores the time when a page is crawled to check whether next time another page of the same site is being crawled it is honoring the intended crawl delay and request rate limits.
Requirements:
PHP 5.0 or higher
License type: GPL - GNU General Public License
Date added: 4 years, 8 months 1 day ago | Last updated: 4 years, 8 months 1 day ago
More popular Web Search
- Google Custom Search 1.0 (2 years, 11 months 5 days ago)
- Yahoo Boss Search 1.0.1 (2 years, 6 months 26 days ago)
- Search Keywords 1.0 (6 years, 11 months 5 days ago)
- Spider Class 1.0 (4 years, 8 months 1 day ago)
- Search and Replace class 1.0 (5 years, 3 months 6 days ago)
This is actually one of the most advanced image hosting script. The imagetize key features: SEO, Ads Ready, Admin
Listing Files
robots_txt
Hot Scripts
- Mysql Ajax Table Editor
- PHP DataGrid
- Console App
- AJAX Login Module
- Form Tools
- Amba XML document
- LimeSurvey
- XLS Export
- ajaxCRUD
- SuperCali PHP Event Calendar
Sponsors
- FREE MySQL Cloud Database Super Simple, Zero-Management MySQL Cloud DB. Use It FREE!
- Pinterest clone Start your own pin board social network today. Get Pintastic and start your own pinterest clone network.
- MYRE Real Estate Software Professional Real Estate solutions by MYREPHP PROGRAMMING!
