Re: Manuals and Spiders

> looking at the huge percentage of visits that come from search robots > [1], I think it would be reasonable to create a robots.txt that prevents > those crawlers to index php.net and its mirrors. > > php.net itself features a full-text search, many mirrors do so, too. PHP > does not take any advantage by being searched by crawlers (at least in > the manuals). > > So I'd propose to create a robots.txt that at least prevents indexing > the manuals. I never thought that this is such a big problem. I beleive many guys search with google and other engines just to find a manual page providing enough information to solve their problems. The site search engines are also not really powerful, thogh there is a huge potential in them, someone just needs to write that frontend... [Which I am going to do anyway, but I need time ;)]. If this really hurts, then it would be nice to disallow robots from the printed pages at least, so half of the indexing would be turned away (there used to be a bug report about this too...). But robots.txt does not provide any wildcard methods, so all the languages need to be manually listed... This probably means, that robots.txt should be generated on rsync.php.net to make it automatic... Goba