Re: Blatantly Evil Question

From:	Jochem Maas	Date:	Fri, 12 Aug 2005 00:20:57 +0000
Subject:	Re: Blatantly Evil Question
References:	1 2 3 4	Groups:	php.general
Request:	Send a blank email to php-general+get-220492@lists.php.net to get a copy of this message

Jasper Bryant-Greene wrote:

Brian Dunning wrote:
On Aug 11, 2005, at 3:44 PM, Evert | Collab wrote:
Use robots.txt 'evil' searchengines will spoof the user-agent string anyway
Can you be more specific about what you mean by "use robots.txt"? I just want to cloak for Google, MSN, and Yahoo. I couldn't care less about what any other search engine (evil or not) does or sees.
robots.txt will not do what you want it to. Just sniff for those robots' User-Agents (Google, MSN and Yahoo all publish their UA strings on their websites, AFAIK) and send different content if it's one of those.

they will hammer you for it eventually - AFAICT all major SEs send out their spiders occasionally with faked user-agent strings - to catch out crap like this. oh and the guy that invented php is a really bigcheese down at yahoo... and he reads this list :-) though I doubt he has the time or desire to chase you personally. I would recommend you don't go down this road. it's bad for your business in the longer term and its bad for the web because your filling it with shite.

Jasper