Re: Manuals and Spiders

On Tue, 22 Apr 2003, Alex Kiesel wrote: > looking at the huge percentage of visits that come from search robots > [1], I think it would be reasonable to create a robots.txt that prevents > those crawlers to index php.net and its mirrors. > > php.net itself features a full-text search, many mirrors do so, too. PHP > does not take any advantage by being searched by crawlers (at least in > the manuals). > > So I'd propose to create a robots.txt that at least prevents indexing > the manuals. > > What do you think? The quality of the current manual search function is quite bad, and I usually just use the url: function of alltheweb.com to search in the PHP manual. Derick -- ------------------------------------------------------------------------- Derick Rethans http://derickrethans.nl/ JDI Media Solutions http://www.jdimedia.nl/ PHP Magazine - PHP Magazine for Professionals http://php-mag.net/ -------------------------------------------------------------------------