Re: Making web spider clever
| From: | Peter Clarke | Date: | Fri, 10 Aug 2001 16:24:48 +0000 |
| Subject: | Re: Making web spider clever | ||
| References: | 1 | Groups: | php.general |
| Request: | Send a blank email to php-general+get-62189@lists.php.net to get a copy of this message | ||
"Phillip Lockwood-Holmes" <phillip@kagoule.co.uk> wrote in message
news:20010810144602.6781.qmail@pb1.pair.com...
> I am working on a basic web spider PHP script running as Shell / CGI on a
> Linux box. At the moment I have the most basic piece of PHP to read in the
> contents of a URL into a variable.
>
> function OpenURL ($URL) {
> global $MaxFileLength;
>
> $SpiderSource = fopen ($URL, "r");
> $SpiderPage = fread ($SpiderSource, $MaxFileLength);
> fclose ($SpiderSource);
>
> return $SpiderPage;
> }
>
> I am going to put in some error trapping on the fopen command and set
> $SpiderPage to "Offline". However sometimes the fopen doesn't error but
> either that or fread take ages. I presume this is due to general slow
> internet/server issues. I therefore want to build in something which will
> timeout and set $SpiderPage to "TimeOut" rather than the script just
getting
> stuck.
>
> Any help would be great. Thanks in advance,
>
>
> Phillip
>
>
You get more flexibilty with fsockopen() [ http://www.php.net/fsockopen ]
allowing for time-outs and sending whatever headers you want. Also fopen()
is often prevented from opening remote files as a security measure.
Here's what I use to collect a web page into a string (with a 15 second
time-out):
$string = external_fileread($url);
function external_fileread($url) {
$content = "";
$urlArray = parse_url($url);
if (!@$urlArray["port"]) $urlArray["port"] = "80";
if (!@$urlArray["path"]) $urlArray["path"] = "/";
if (@$urlArray["query"]) $urlArray["path"] .= "?$urlArray[query]";
$dump = "GET $urlArray[path] HTTP/1.0\r\n";
$dump .= "Accept: */*\r\n";
$dump .= "Host: $urlArray[host]\r\n";
$dump .= "User-Agent: PHP script\r\n";
$dump .= "Connection: close\r\n";
$dump .= "\r\n";
$socket = fsockopen($urlArray["host"], $urlArray["port"], &$errnum,
&$errstr, 15);
if (!@$socket) {
return 0;
} else {
fputs($socket, $dump);
while($str = fgets($socket, 1024)){
if ($headers_finished){
$content .= $str;
}
if (($str == "\n")||($str == "\r\n")){ // headers finish with a blank
line
$headers_finished = true;
}
}
fclose($socket);
return $content;
}
return 0;
}
Peter