Re: Robots.txt file restricting msnbot at http://news.php.net/robots.txt
| From: | Hannes Magnusson | Date: | Sat, 09 Aug 2008 10:35:54 +0000 |
| Subject: | Re: Robots.txt file restricting msnbot at http://news.php.net/robots.txt | ||
| References: | 1 | Groups: | php.webmaster |
| Request: | Send a blank email to php-webmaster+get-2357@lists.php.net to get a copy of this message | ||
On Sat, Aug 9, 2008 at 02:13, Amy Wilcox (Murphy & Associates)
<v-amwilc@microsoft.com> wrote:
> We would be pleased if you edit your robots.txt file to allow our search
> engine to index your content which additionally will increase traffic to
> your site via our search results.
That is exactly why we added that file. We don't want spiders indexing
that archive.
If you want to index our mailinglist archives then feel free to do so,
just not via search engine spider on our news frontend.
-Hannes