Re: Fw: Spoofing user_agent
| From: | Thiago H. Pojda | Date: | Tue, 30 Nov 2010 02:19:14 +0000 |
| Subject: | Re: Fw: Spoofing user_agent | ||
| References: | 1 2 3 4 5 6 7 8 9 10 11 | Groups: | php.general |
| Request: | Send a blank email to php-general+get-309762@lists.php.net to get a copy of this message | ||
Quit top posting.
On Mon, Nov 29, 2010 at 9:55 PM, Ron Piggott <ron.piggott@actsministries.org
> wrote:
>
> My issue with the user agent is unresolved. I need to do more research to
> see how AWSTATS distinguishes between a robot crawling the site and a web
> page user and set the user-agent accordingly.
>
Ron,
AWSTATS probably users a knowledge base for known bots, I'm not sure. If
that's the case, you can just set your User-Agent to a known and see how
that goes.
Look for Googlebot, Majestic, Ask.com (now dead - probably a good pick),
MSNBot here: http://www.user-agents.org/
As for setting the User-Agent in your request, I like to use this cUrl
snippet (based on a note at curl's manual page):
<?php
$sUrl = 'www.example.com/';
$sUserAgent = 'Googlebot/2.1 (+http://www.googlebot.com/bot.html)';
$hCurl = curl_init();
curl_setopt ($hCurl, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt ($hCurl, CURLOPT_URL, $sUrl);
curl_setopt ($hCurl, CURLOPT_CONNECTTIMEOUT, 120);
curl_setopt ($hCurl, CURLOPT_TIMEOUT, 120);
curl_setopt ($hCurl, CURLOPT_USERAGENT, $sUserAgent);
$sContent = curl_exec($hCurl);
?>
Cheers,
Thiago Henrique Pojda
+55 41 8856-7925