Re: Manuals and Spiders

Just let Stig join into our conversation, as we are talking about him and Alltheweb ;) Stig, please read on... Goba > > > looking at the huge percentage of visits that come from search robots > > > [1], I think it would be reasonable to create a robots.txt that prevents > > > those crawlers to index php.net and its mirrors. > > > > > > php.net itself features a full-text search, many mirrors do so, too. PHP > > > does not take any advantage by being searched by crawlers (at least in > > > the manuals). > > > > > > So I'd propose to create a robots.txt that at least prevents indexing > > > the manuals. > > > > > > What do you think? > > > > The quality of the current manual search function is quite bad, and I > > usually just use the url: function of alltheweb.com to search in the > > PHP manual. > > Well, Stig said that setting up search.php.net at alltheweb > would not be a huge task (but that was more than one and > a half a year ago)... He said at the intl. conf last year to > me that it is doable... But nothing happened ;(( I would really > be happy to see an alltheweb powered central search engine > for PHP.net with much more features then the current one... > > Goba