Making web spider clever
| From: | Phillip Lockwood-Holmes | Date: | Fri, 10 Aug 2001 14:48:50 +0000 |
| Subject: | Making web spider clever | ||
| Groups: | php.general | ||
| Request: | Send a blank email to php-general+get-62177@lists.php.net to get a copy of this message | ||
I am working on a basic web spider PHP script running as Shell / CGI on a
Linux box. At the moment I have the most basic piece of PHP to read in the
contents of a URL into a variable.
function OpenURL ($URL) {
global $MaxFileLength;
$SpiderSource = fopen ($URL, "r");
$SpiderPage = fread ($SpiderSource, $MaxFileLength);
fclose ($SpiderSource);
return $SpiderPage;
}
I am going to put in some error trapping on the fopen command and set
$SpiderPage to "Offline". However sometimes the fopen doesn't error but
either that or fread take ages. I presume this is due to general slow
internet/server issues. I therefore want to build in something which will
timeout and set $SpiderPage to "TimeOut" rather than the script just getting
stuck.
Any help would be great. Thanks in advance,
Phillip