RE: Google crawl...

Fwiw, I spent some time talking to the googlebot team about this... They said that googlebot was intentionally dumb as far as dealing with certain mimetypes to cache. However php.net has a strange and complicated mirror system with distinct urls - which confuses google and makes it believe that they are individually different sites. I have no problem with dissallowing paths: I had spent some time figuring out what needed dissallowing, and promptly had no time to implement it ... :) - james -- James Cox (Editor) :: james.cox@apress.com :: http://www.apress.com/ The Experts Voice (tm) [After the briefing about a state dinner] C.J. Cregg: Oh, boy, I like it when "In Style" magazine is issued press credentials. "Mirabella" wanted to know what wine is served with the fish course. So it's a good thing I went to school for 22 years. Josh Lyman: What wine are we-- C.J. Cregg: It's wine, you'll drink it! > -----Original Message----- > From: Anders Olausson [mailto:spacedump@burken.nu] > Sent: 31 October 2003 15:04 > To: Spyros Ioakim; 'Gabor Hojtsy' > Cc: webmaster@php.net > Subject: Re: Google crawl... > > Hi, > > I just looked into it and found out that google eats some > bandwidth here too.. > > Month total transfer/google transfer > July 7.81GB/1.27GB > August 14.01GB/5.36GB > September 8.72GB/7.13GB > October 8.62GB/5.69GB > > So, I welcome the change in robots.txt. :) > > Regards > Anders - se2.php.net > ----- Original Message ----- > From: "Spyros Ioakim" <sioakim@ace-hellas.gr> > To: "'Gabor Hojtsy'" <gabor@hojtsy.hu>; "Ioakim Spyros" > <sioakim@ace-hellas.gr> > Cc: <webmaster@php.net> > Sent: Friday, October 31, 2003 11:08 PM > Subject: RE: Google crawl... > > > > I want to agree with the disallow rule. > > Somebody else was also mentioning the same problem with > heavy crawling (I > > think the admin of uk2....) > > > > If you take a look in the gr2 stat page there is a "Top 10 > of 24635 Total > > URLs By Kbytes". > > The total in there is 19GB out of the total 22GB. > > > > Google has crawled about 18GB. So without analysing too > much I would say > > that about 16GB were caused by google downloading the > distribution files > > again and again... > > > > Maybe other admins should also check this out and give us > some feedback > but > > I would suggest that /distributions be disallowed as there > is nothing of > > interested for a search engine in there... > > > > Spyros > > > > -----Original Message----- > > From: Gabor Hojtsy [mailto:gabor@hojtsy.hu] > > Sent: Friday, October 31, 2003 4:52 PM > > To: Ioakim Spyros > > Cc: webmaster@php.net > > Subject: Re: Google crawl... > > > > > I was looking at the stats and noticed that altough 64% > of the hits > > > are from Greek sites only 8% is Greek traffic... > > > > > > So I looked into it and saw that 1.78 GB was transfered > to greek sites > > > and 18.67GB to US... > > > > > > As I can assume most of it should be Google... > > > Check out... > > > http://gr2.php.net/stats/usage_200310.php > > > > We can safely add a disallow rule for /distributions to the > robots.txt > IMHO. > > Any objections? > > > > > Does anyone know what does Google has to crawl the .gz > and .zip files > > > also? > > > > It might crawl the contents of the gz and zip files... > > > > Goba > > > >