Re: cvs: docweb /scripts checkent.php

On Wednesday 19 January 2005 22:48, Nuno Lopes wrote: > >> + fputs($fp, "HEAD {$url['path']} HTTP/1.0\r\nHost: > >> {$url['host']}\r\nConnection: close\r\nUser-agent: Mozilla/5.0 (X11; U; > >> Linux i686; en-US; rv:1.7.5) Gecko/20041207 Firefox/1.0\r\n\r\n"); > > > > Is this really necessary? > > I agree with the sleep, however, faking a user agent is bad form. Is this > > problem theoretical, or does this script really not work with a > > User-Agent like "PHP DocWeb Link Crawler (http://doc.php.net)" ? > > > > We also need to respect robots.txt (consider a denied request > > "uncheckable"). > > I think that sleeping 7 seconds is too much. The script will take about 1 > hour to run. Do we need to sleep() so many time? Indeed 7 seconds is long.. but still I've seen sites blocking the script.. maybe, after some testing, a few seconds (2 or 3), would be enough. I'm going to bed now, and I might only be able to get some free time on friday :( > > The user agent is necessary because I've seen one server rejecting the > request because of the non-existence of the user agent (!). > I think we don't need to over-complicate things, requesting the robots.txt > file, etc... Using a generic UA is enough. a robots.txt is used when you don't want a search engine crawler indexing your files.. I don't see a search engine indexer in this script :) > > > And last: this needs to be templated at the very least. I suggest we dump > > the data into an SQLite DB (via cron, as it runs now), and recall the > > data using a templated web page. > > (I realize this last part will take time.. I'm not saying it needs to be > > done NOW.. just needs to be done). > > This script probably doesn't have interest for translations teams. Maybe we > could restrict it to 'all' and 'en' languages and leave it as-is. also, there is not much to translate.. and I asume only a few people will really 'use' the script.. Vincent > > > Nuno