Re: Google getting over excited...
| From: | Vahan Yerkanian | Date: | Mon, 08 Sep 2003 18:11:00 +0000 |
| Subject: | Re: Google getting over excited... | ||
| References: | 1 | Groups: | php.mirrors |
| Request: | Send a blank email to php-mirrors+get-20904@lists.php.net to get a copy of this message | ||
Hello All,
Yeah, the same is here at am.php.net
Most of the traffic for the past 2-3 weeks are from google and altavista. Isn't there a robots.txt to control excessive indexing of the mirror?
regards,
Vahan Yerkanian
Chris Burton wrote:
Hi, Has anyone else noticed google are getting abit carried away indexing the mirrors? So far this month googlebot.com have pulled over 8GB (in 7 days from uk2.php.net) which is on avg more than people using the mirror use per day. The amount is also twice that used last month in total from googlebot. Any idea why the sudden increase in crawling recently from googlebot? Thanks, Chris Burton