RE: Google crawl...

google is 7% of my visitor traffic in my mirror as well... i don't like it... http://il.php.net/stats/usage_200310.php On Sat, 1 Nov 2003, Spyros Ioakim wrote: > I want to agree with the disallow rule. > Somebody else was also mentioning the same problem with heavy crawling (I > think the admin of uk2....) > > If you take a look in the gr2 stat page there is a "Top 10 of 24635 Total > URLs By Kbytes". > The total in there is 19GB out of the total 22GB. > > Google has crawled about 18GB. So without analysing too much I would say > that about 16GB were caused by google downloading the distribution files > again and again... > > Maybe other admins should also check this out and give us some feedback but > I would suggest that /distributions be disallowed as there is nothing of > interested for a search engine in there... > > Spyros > > -----Original Message----- > From: Gabor Hojtsy [mailto:gabor@hojtsy.hu] > Sent: Friday, October 31, 2003 4:52 PM > To: Ioakim Spyros > Cc: webmaster@php.net > Subject: Re: Google crawl... > > > I was looking at the stats and noticed that altough 64% of the hits > > are from Greek sites only 8% is Greek traffic... > > > > So I looked into it and saw that 1.78 GB was transfered to greek sites > > and 18.67GB to US... > > > > As I can assume most of it should be Google... > > Check out... http://gr2.php.net/stats/usage_200310.php > > We can safely add a disallow rule for /distributions to the robots.txt IMHO. > Any objections? > > > Does anyone know what does Google has to crawl the .gz and .zip files > > also? > > It might crawl the contents of the gz and zip files... > > Goba > -- Best regards, Shimi ---- "Outlook is a massive flaming horrid blatant security violation, which also happens to be a mail reader." -=The best way to accelerate a Windows machine is at 9.81 m/s^2=- "Windows is a 32-bit port of a 16-bit GUI for a 8-bit OS on a 4-bit CPU made by a 2-bit company that can't stand 1 bit of competition."