RE: Google crawl...

I want to agree with the disallow rule. Somebody else was also mentioning the same problem with heavy crawling (I think the admin of uk2....) If you take a look in the gr2 stat page there is a "Top 10 of 24635 Total URLs By Kbytes". The total in there is 19GB out of the total 22GB. Google has crawled about 18GB. So without analysing too much I would say that about 16GB were caused by google downloading the distribution files again and again... Maybe other admins should also check this out and give us some feedback but I would suggest that /distributions be disallowed as there is nothing of interested for a search engine in there... Spyros -----Original Message----- From: Gabor Hojtsy [mailto:gabor@hojtsy.hu] Sent: Friday, October 31, 2003 4:52 PM To: Ioakim Spyros Cc: webmaster@php.net Subject: Re: Google crawl... > I was looking at the stats and noticed that altough 64% of the hits > are from Greek sites only 8% is Greek traffic... > > So I looked into it and saw that 1.78 GB was transfered to greek sites > and 18.67GB to US... > > As I can assume most of it should be Google... > Check out... http://gr2.php.net/stats/usage_200310.php We can safely add a disallow rule for /distributions to the robots.txt IMHO. Any objections? > Does anyone know what does Google has to crawl the .gz and .zip files > also? It might crawl the contents of the gz and zip files... Goba