Re: Manuals and Spiders

On Apr 22, 2003, Gabor Hojtsy wrote: > > looking at the huge percentage of visits that come from search robots > > [1], I think it would be reasonable to create a robots.txt that prevents > > those crawlers to index php.net and its mirrors. > > > > php.net itself features a full-text search, many mirrors do so, too. PHP > > does not take any advantage by being searched by crawlers (at least in > > the manuals). > > > > So I'd propose to create a robots.txt that at least prevents indexing > > the manuals. > > I never thought that this is such a big problem. Don't misunderstand. This was just a proposal, I can perfectly live with google indexing those pages. Perhaps one can use those search engines to get load off of php.net and move it to some mirrors (IIRC someone said that this would help php.net), by only letting mirrors be indexes for example. > If this really hurts, then it would be nice to > disallow robots from the printed pages at least, > so half of the indexing would be turned away (there > used to be a bug report about this too...). But > robots.txt does not provide any wildcard methods, > so all the languages need to be manually listed... > This probably means, that robots.txt should be > generated on rsync.php.net to make it automatic... That would at least be reasonable :) If it makes too much work, just let it the way it is... -Alex