+ fputs($fp, "HEAD {$url['path']} HTTP/1.0\r\nHost: {$url['host']}\r\nConnection: close\r\nUser-agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041207 Firefox/1.0\r\n\r\n");
Is this really necessary?
I agree with the sleep, however, faking a user agent is bad form. Is this problem theoretical, or does this script really not work with a User-Agent like "PHP DocWeb Link Crawler (http://doc.php.net)" ?
We also need to respect robots.txt (consider a denied request "uncheckable").
And last: this needs to be templated at the very least. I suggest we dump the data into an SQLite DB (via cron, as it runs now), and recall the data using a templated web page.
(I realize this last part will take time.. I'm not saying it needs to be done NOW.. just needs to be done).
S