Re: some ideas for open discussion

From: Date: Fri, 23 May 2003 20:01:17 +0000
Subject: Re: some ideas for open discussion
References: 1  Groups: php.mirrors 
Request: Send a blank email to php-mirrors+get-17976@lists.php.net to get a copy of this message
Oh, so blocking bots from indexing shortcuts. That is fine with me. In case you can point to some good resource on how to identify bots ;)
Uh, a simple robots.txt rule should do the trick. But I am not all that convinced we should block them. Why not let it find our nice short urls, especially if we reduce it to a single request as you suggested.
OK, I am not against showing the searh engines the short URLs ;) That may have a problem of more irrelevant results in searches though. Pages listed more then once with the same content (on www search engines). Our shortcut URLs are also quite diverse, I don't know of any way to block robots from just those with a robots.txt rule. :( Goba

« previous php.mirrors (#17976) next »