+ fputs($fp, "HEAD {$url['path']} HTTP/1.0\r\nHost: {$url['host']}\r\nConnection: close\r\nUser-agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.7.5) Gecko/20041207 Firefox/1.0\r\n\r\n");
Is this really necessary?
I agree with the sleep, however, faking a user agent is bad form. Is this problem theoretical, or does this script really not work with a User-Agent like "PHP DocWeb Link Crawler (http://doc.php.net)" ?
We also need to respect robots.txt (consider a denied request "uncheckable").
I think that sleeping 7 seconds is too much. The script will take about 1 hour to run. Do we need to sleep() so many time?
The user agent is necessary because I've seen one server rejecting the request because of the non-existence of the user agent (!).
I think we don't need to over-complicate things, requesting the robots.txt file, etc... Using a generic UA is enough.
And last: this needs to be templated at the very least. I suggest we dump the data into an SQLite DB (via cron, as it runs now), and recall the data using a templated web page.
(I realize this last part will take time.. I'm not saying it needs to be done NOW.. just needs to be done).
This script probably doesn't have interest for translations teams. Maybe we could restrict it to 'all' and 'en' languages and leave it as-is.
Nuno