<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
<head>
<title>Documentation for method:
PHPCrawler::obeyRobotsTxt()</title>
<meta name="keywords" content="framework, API, manual, class reference, classreference, documentation" />
<meta name="description" content="The class reference contains the detailed description of how to use every class, method, and property." />
<link rel="stylesheet" type="text/css" media="screen" href="style.css">
<script name="javascript">
function show_hide_examples(mode)
{
if (document.getElementById("examples").style.display == "none")
{
document.getElementById("examples").style.display = "";
}
else
{
document.getElementById("examples").style.display = "none";
}
}
</script>
</head>
<body>
<div id="outer">
<h1 id="head">
<span>Method:
PHPCrawler::obeyRobotsTxt()</span>
</h1>
<h2 id="head">
<span><a href="overview.html"><< Back to class-overview</a></span>
</h2>
<br>
<!--<?php include("google_code.php"); ?> -->
<div id="docframe">
<div id="section">
Decides whether the crawler should parse and obey robots.txt-files.
</div>
<div id="section">
<b>Signature:</b>
<p id="signature">
public obeyRobotsTxt($mode)
</p>
</div>
<div id="section">
<b>Parameters:</b>
<p>
<table id="param_list">
<tr><td id="paramname" width="1%"><b>$mode</b> </td><td width="1%"><i><i>bool</i></i> </td><td width="*">Set to TRUE if you want the crawler to obey robots.txt-files.</td></tr>
</table>
</p>
</div>
<div id="section">
<b>Returns:</b>
<p>
<table id="param_list">
<tr> <td width="1%"><i><i>bool</i></i> </td> <td width="*"></td></tr>
</table>
</p>
</div>
<div id="section">
<b>Description:</b>
<p>
If this is set to TRUE, the crawler looks for a robots.txt-file for every host that sites or files should be received<br>from during the crawling process. If a robots.txt-file for a host was found, the containig directives appliying to the<br>useragent-identification of the cralwer<br>("PHPCrawl" or manually set by calling <a href="method_detail_tpl_method_setUserAgentString.htm" class="inline">setUserAgentString()</a>) will be obeyed.<br><br>The default-value is FALSE (for compatibility reasons).<br><br>Pleas note that the directives found in a robots.txt-file have a higher priority than other settings made by the user.<br>If e.g. <a href="method_detail_tpl_method_addFollowMatch.htm" class="inline">addFollowMatch</a>("#http://foo\.com/path/file\.html#") was set, but a directive in the robots.txt-file of the host<br>foo.com says "Disallow: /path/", the URL http://foo.com/path/file.html will be ignored by the crawler anyway.
</p>
</div>
</div>
<div id="footer">Docs created with <a href="http://phpclassview.cuab.de" target="_parent">PhpClassView</a></div>
</div>
</body>
</html>