Re: Blatantly Evil Question

From:	Jochem Maas	Date:	Fri, 12 Aug 2005 00:59:23 +0000
Subject:	Re: Blatantly Evil Question
References:	1 2 3 4 5 6	Groups:	php.general
Request:	Send a blank email to php-general+get-220495@lists.php.net to get a copy of this message

Philip Hallstrom wrote:

robots.txt will not do what you want it to. Just sniff for those robots' User-Agents (Google, MSN and Yahoo all publish their UA strings on their websites, AFAIK) and send different content if it's one of those.
they will hammer you for it eventually - AFAICT all major SEs send out their spiders occasionally with faked user-agent strings - to catch out crap like this.
google adsense won't. I explicity asked them about this. Well, what I asked was that if I had a password protected area, could I allow them access to spider the content so that normal users could see the ads. I told them the layout would be different, but the content the same. They said that was fine.

but you didn't ask - 'heh is it okay to fill my public page with SEO crud but only if a spider comes round' they might just take a different view on that :-)

2cents.