Re: some ideas for open discussion

On Friday 23 May 2003 21:47, Rasmus Lerdorf <rasmus@lerdorf.com> wrote: > On Fri, 23 May 2003, Gabor Hojtsy wrote: > > Oh, so blocking bots from indexing shortcuts. That is fine with me. In > > case you can point to some good resource on how to identify bots ;) > > Uh, a simple robots.txt rule should do the trick. But I am not all that > convinced we should block them. Why not let it find our nice short urls, > especially if we reduce it to a single request as you suggested. > > -Rasmus Thinking about something now... maybe my blocking bots wasn't a good idea... but when the bot will visit a shortcut that doesn't exists in the manual it will do the htdig. so blocking bots on /search.php should help. there is no need to index search results in a search engine. I don't see the point in that.. just an idea.. .: Allowee