Re: cvs: docweb /scripts checkent.php

On Wednesday 19 January 2005 23:04, Sean Coates wrote: > Nuno Lopes wrote: > > I think that sleeping 7 seconds is too much. The script will take about > > 1 hour to run. Do we need to sleep() so many time? > > Ah, now that you point that out (and I think of it, more) I agree. > Additionally, most links are not on the same server, correct? Therefore > different servers wouldn't even know about our sleep. yep, and looking a bit deeper into the output only seems to show that apache.org is using this ddos protection. I think a simple check will allow us to sleep() only for the apache.org site. and atm there are no servers with an internal server error url.filepro http://www.fptech.com/ url.iicm http://www.iicm.edu/ both work fine, but a 'HEAD /' is not properly handled maybe we need to add them in the script to make another request so that we don't get false data (GET / seems to return a 200 status) the pecl.php.net/get/ also isn't really not found, I think it should be 'hidden' from the list And I also noticed that some sites redirect there users to other subdomains.. like www.netcraft.com goes to news.netcraft.com we just can't change this, IMO.. it could always be changed back to www. by netcraft. same goes for google we link to a news group archive, it gets redirected to groups-beta.google.. but things like www.sourceforge.net that goes to sourceforge.net (libssh2) don't seem to be a problem, sf.net has been doing this for years The goal of this script should be to not list anything, making all urls correct :) I've now just wrote down my brainstorm about this.. I'm going for some exams now, they end on friday. Vincent > > > The user agent is necessary because I've seen one server rejecting the > > request because of the non-existence of the user agent (!). > > Right. Omitting UA is bad. Sending our own UA is good -- a webserver > that has a whitelist of UAs is going to deny a significant part of the > internet population, anyway. > > > I think we don't need to over-complicate things, requesting the > > robots.txt file, etc... Using a generic UA is enough. > > Requesting (and honoring) robots.txt, IMO, is necessary. We're running a > non-human user agent, here. But I don't see any hard rule about this on > > the W3C site, so.. <shrug> (-: > > This script probably doesn't have interest for translations teams. Maybe > > we could restrict it to 'all' and 'en' languages and leave it as-is. > > You're right. > "all" is "en", BTW.. unless the default language ever changes (which I > doubt). > > ... Vincent's reply came in at the end of me writing this, but his > points were very similar. > > S