Re: Detecting Bots...
| From: | Evan Nemerson | Date: | Tue, 10 Jun 2003 03:50:15 +0000 |
| Subject: | Re: Detecting Bots... | ||
| References: | 1 | Groups: | php.general |
| Request: | Send a blank email to php-general+get-150845@lists.php.net to get a copy of this message | ||
This should be helpful:
http://www.robotstxt.org/wc/active/all.txt
On Monday 09 June 2003 07:36 pm, Steven Kallstrom wrote:
> Dear List,
>
> So $_SERVER['HTTP_REFERER'] gives us the User Agent... is there anyway
> to determine if in general the User-Agent is a bot, or do I have to
> check for a specific bot... I want to strip out all GET information
> from the URL so that bots can crawl my site session free...
>
> Thanks,
>
> SJK