detecting humans using the user-agent
| From: | Daryl Meese | Date: | Wed, 28 Apr 2004 13:08:17 +0000 |
| Subject: | detecting humans using the user-agent | ||
| Groups: | php.general | ||
| Request: | Send a blank email to php-general+get-184897@lists.php.net to get a copy of this message | ||
Hello all,
I have included a function named isHuman below. isHuman uses three checks on the HTTP_USER_AGENT to
detect whether a page request has come from a human or a spider/robot. First it looks for a series
of strings that are common in human browser user-agents then, if it found one of those strings,
performs a check for common spider/robot user agents that would pass the first test. It could also
perform checks for ip addresses that are known to be spiders/robots.
We are going to use this function to filter hits on an advertising system and limit what can view
the contact information presented in our "contact us" sections but I am sure there are
many more uses. For example, the archives for this list could hide emails completely (not even show
meAThereDOTcom) if the user-agent doesn't pass the test.
So, Why did I post it?
1. To share it with all you great people;>
2. To correct errors and ommissions from the goodWords, badWords, exactMatch arrays in the
function.
3. To get feedback on how it might be improved.
I will take all comments and suggestions off list because there is really no reason to bog anyone
else down. If the function changes I will email a new copy to anyone requesting so and anyone that
has provided feedback. If that list of people gets large I will repost the function to the list.
Thanks,
Daryl Meese
Magia Software
function isHuman($default = 1)
{
$user_agent_lc = strtolower($_SERVER['HTTP_USER_AGENT']);
// Set the isHuman var to the default if it is valid, else default to true
if ($default === 0 || $default === 1)
{
$isHuman = $default;
}
else
{
$isHuman = 1;
}
$goodWords = array ('anonymized','aol','explorer 6','field
blocked
by','galeon','gecko','icab','kmeleon','konqueror','lotus-notes','lynx','mozilla','msie','netscape','omniweb','opera','webtv');
for ($x = 0; $x < count($goodWords); $x++)
{
if (ereg("$goodWords[$x]","$user_agent_lc"))
{ // Got a match - mark as good and move on
$isHuman = 1;
break;
}
}
// if it is marked as good, make sure it's not just playing games
if ($isHuman == 1)
{
// exact matches first - the elements in this array will be compared for exact matching
$exactMatch = array ('mozilla');
for ($x = 0; $x < count($exactMatch); $x++)
{
if ($user_agent_lc == $exactMatch[$x])
{
return 0;
}
}
// now check for tail-tail signs of fraudulent browser entry
$badWords = array
('\.com','\.de','\.mail','\.net','\.nl','\.org','@','amsterdam',
'ask
jeeves','askjeeves','avirt','bordermanager','check&get',
'dreamcast','email','ezresult','fastcrawler','fetch','googlebot',
'grub','hotbar','httrack','ics','indy','indy
library','links2go',
'msiecrawler','muscatferret','netnose-crawler','netpliance','newt',
'noydb','obot','openbot','piranha','psbot','shark','slurp','slurp',
'snap','spider','t-h-u-n-d-e-r-s-t-o-n-e','transmission','vagabond',
'webinator','winhttp','yahoo');
for ($x = 0; $x < count($badWords); $x++)
{
if (ereg("$badWords[$x]","$user_agent_lc"))
{ // Got a match - mark as good and move on
return 0;
}
}
}
return $isHuman;
}