Re: Google crawl...

Hi, I just looked into it and found out that google eats some bandwidth here too.. Month total transfer/google transfer July 7.81GB/1.27GB August 14.01GB/5.36GB September 8.72GB/7.13GB October 8.62GB/5.69GB So, I welcome the change in robots.txt. :) Regards Anders - se2.php.net ----- Original Message ----- From: "Spyros Ioakim" <sioakim@ace-hellas.gr> To: "'Gabor Hojtsy'" <gabor@hojtsy.hu>; "Ioakim Spyros" <sioakim@ace-hellas.gr> Cc: <webmaster@php.net> Sent: Friday, October 31, 2003 11:08 PM Subject: RE: Google crawl... > I want to agree with the disallow rule. > Somebody else was also mentioning the same problem with heavy crawling (I > think the admin of uk2....) > > If you take a look in the gr2 stat page there is a "Top 10 of 24635 Total > URLs By Kbytes". > The total in there is 19GB out of the total 22GB. > > Google has crawled about 18GB. So without analysing too much I would say > that about 16GB were caused by google downloading the distribution files > again and again... > > Maybe other admins should also check this out and give us some feedback but > I would suggest that /distributions be disallowed as there is nothing of > interested for a search engine in there... > > Spyros > > -----Original Message----- > From: Gabor Hojtsy [mailto:gabor@hojtsy.hu] > Sent: Friday, October 31, 2003 4:52 PM > To: Ioakim Spyros > Cc: webmaster@php.net > Subject: Re: Google crawl... > > > I was looking at the stats and noticed that altough 64% of the hits > > are from Greek sites only 8% is Greek traffic... > > > > So I looked into it and saw that 1.78 GB was transfered to greek sites > > and 18.67GB to US... > > > > As I can assume most of it should be Google... > > Check out... http://gr2.php.net/stats/usage_200310.php > > We can safely add a disallow rule for /distributions to the robots.txt IMHO. > Any objections? > > > Does anyone know what does Google has to crawl the .gz and .zip files > > also? > > It might crawl the contents of the gz and zip files... > > Goba >