On Wednesday 19 January 2005 23:04, Sean Coates wrote:
Nuno Lopes wrote:
I think that sleeping 7 seconds is too much. The script will take about
1 hour to run. Do we need to sleep() so many time?
Ah, now that you point that out (and I think of it, more) I agree.
Additionally, most links are not on the same server, correct? Therefore
different servers wouldn't even know about our sleep.
yep, and looking a bit deeper into the output only seems to show that
apache.org is using this ddos protection.
I think a simple check will allow us to sleep() only for the apache.org site.
IMHO, the sleep should be done only when handling multiple urls from the same host. Thats an easy thing to do.
and atm there are no servers with an internal server error
url.filepro http://www.fptech.com/
url.iicm http://www.iicm.edu/
both work fine, but a 'HEAD /' is not properly handled
maybe we need to add them in the script to make another request so that we
don't get false data (GET / seems to return a 200 status)
Thats strange, because the HTTP RFC clearly states that the servers should return the same headers with both HEAD and GET.
I'll investigate this further...
the pecl.php.net/get/ also isn't really not found, I think it should be
'hidden' from the list
Yep, this one could be hidden.
And I also noticed that some sites redirect there users to other subdomains..
like www.netcraft.com goes to news.netcraft.com
we just can't change this, IMO.. it could always be changed back to www. by
netcraft.
same goes for google
we link to a news group archive, it gets redirected to groups-beta.google..
Yes. And there others that redirect to /index.php. those shouldn't be changed, of course.
but things like www.sourceforge.net that goes to sourceforge.net (libssh2)
don't seem to be a problem, sf.net has been doing this for years
The goal of this script should be to not list anything, making all urls
correct :)
I'll add a condition to output "No bogus URLs found" :)
I've now just wrote down my brainstorm about this..
I'm going for some exams now, they end on friday.
Vincent
So, good luck ;)
Nuno
P.S.: I'll finish the script on the weekend (implement relative URLs, sleep() only when necesssary, etc..)