Manuals and Spiders

Hi, looking at the huge percentage of visits that come from search robots [1], I think it would be reasonable to create a robots.txt that prevents those crawlers to index php.net and its mirrors. php.net itself features a full-text search, many mirrors do so, too. PHP does not take any advantage by being searched by crawlers (at least in the manuals). So I'd propose to create a robots.txt that at least prevents indexing the manuals. What do you think? Cheers, -Alex [1] At this moment, I can grep from my logs: > php3:~ # grep 'www.googlebot.com' /var/log/httpd/php3.de/access_log | wc -l > 989661 > php3:~ # wc -l /var/log/httpd/php3.de/access_log > 9514131 /var/log/httpd/php3.de/access_log This is roughly 10% of all traffic.