Re: Blatantly Evil Question
| From: | Jasper Bryant-Greene | Date: | Thu, 11 Aug 2005 23:13:10 +0000 |
| Subject: | Re: Blatantly Evil Question | ||
| References: | 1 2 3 | Groups: | php.general |
| Request: | Send a blank email to php-general+get-220488@lists.php.net to get a copy of this message | ||
Brian Dunning wrote:
On Aug 11, 2005, at 3:44 PM, Evert | Collab wrote:robots.txt will not do what you want it to. Just sniff for those robots' User-Agents (Google, MSN and Yahoo all publish their UA strings on their websites, AFAIK) and send different content if it's one of those. JasperUse robots.txt 'evil' searchengines will spoof the user-agent string anywayCan you be more specific about what you mean by "use robots.txt"? I just want to cloak for Google, MSN, and Yahoo. I couldn't care less about what any other search engine (evil or not) does or sees.