Making web spider clever

From: Date: Fri, 10 Aug 2001 14:48:50 +0000
Subject: Making web spider clever
Groups: php.general 
Request: Send a blank email to php-general+get-62177@lists.php.net to get a copy of this message
I am working on a basic web spider PHP script running as Shell / CGI on a Linux box. At the moment I have the most basic piece of PHP to read in the contents of a URL into a variable. function OpenURL ($URL) { global $MaxFileLength; $SpiderSource = fopen ($URL, "r"); $SpiderPage = fread ($SpiderSource, $MaxFileLength); fclose ($SpiderSource); return $SpiderPage; } I am going to put in some error trapping on the fopen command and set $SpiderPage to "Offline". However sometimes the fopen doesn't error but either that or fread take ages. I presume this is due to general slow internet/server issues. I therefore want to build in something which will timeout and set $SpiderPage to "TimeOut" rather than the script just getting stuck. Any help would be great. Thanks in advance, Phillip

« previous php.general (#62177) next »