Re: Google crawl...
| From: | Chris Burton | Date: | Fri, 31 Oct 2003 17:32:45 +0000 |
| Subject: | Re: Google crawl... | ||
| References: | 1 2 | Groups: | php.mirrors |
| Request: | Send a blank email to php-mirrors+get-21726@lists.php.net to get a copy of this message | ||
> > As I can assume most of it should be Google...
> > Check out... http://gr2.php.net/stats/usage_200310.php
>
> We can safely add a disallow rule for /distributions to the robots.txt
> IMHO. Any objections?
This is something I would like to see, I'm sure it would help with some of
the +20GB that google uses each month.
ChrisB.