detecting humans using the user-agent

From: Date: Wed, 28 Apr 2004 13:08:17 +0000
Subject: detecting humans using the user-agent
Groups: php.general 
Request: Send a blank email to php-general+get-184897@lists.php.net to get a copy of this message
Hello all, I have included a function named isHuman below. isHuman uses three checks on the HTTP_USER_AGENT to detect whether a page request has come from a human or a spider/robot. First it looks for a series of strings that are common in human browser user-agents then, if it found one of those strings, performs a check for common spider/robot user agents that would pass the first test. It could also perform checks for ip addresses that are known to be spiders/robots. We are going to use this function to filter hits on an advertising system and limit what can view the contact information presented in our "contact us" sections but I am sure there are many more uses. For example, the archives for this list could hide emails completely (not even show meAThereDOTcom) if the user-agent doesn't pass the test. So, Why did I post it? 1. To share it with all you great people;> 2. To correct errors and ommissions from the goodWords, badWords, exactMatch arrays in the function. 3. To get feedback on how it might be improved. I will take all comments and suggestions off list because there is really no reason to bog anyone else down. If the function changes I will email a new copy to anyone requesting so and anyone that has provided feedback. If that list of people gets large I will repost the function to the list. Thanks, Daryl Meese Magia Software function isHuman($default = 1) { $user_agent_lc = strtolower($_SERVER['HTTP_USER_AGENT']); // Set the isHuman var to the default if it is valid, else default to true if ($default === 0 || $default === 1) { $isHuman = $default; } else { $isHuman = 1; } $goodWords = array ('anonymized','aol','explorer 6','field blocked by','galeon','gecko','icab','kmeleon','konqueror','lotus-notes','lynx','mozilla','msie','netscape','omniweb','opera','webtv'); for ($x = 0; $x < count($goodWords); $x++) { if (ereg("$goodWords[$x]","$user_agent_lc")) { // Got a match - mark as good and move on $isHuman = 1; break; } } // if it is marked as good, make sure it's not just playing games if ($isHuman == 1) { // exact matches first - the elements in this array will be compared for exact matching $exactMatch = array ('mozilla'); for ($x = 0; $x < count($exactMatch); $x++) { if ($user_agent_lc == $exactMatch[$x]) { return 0; } } // now check for tail-tail signs of fraudulent browser entry $badWords = array ('\.com','\.de','\.mail','\.net','\.nl','\.org','@','amsterdam', 'ask jeeves','askjeeves','avirt','bordermanager','check&get', 'dreamcast','email','ezresult','fastcrawler','fetch','googlebot', 'grub','hotbar','httrack','ics','indy','indy library','links2go', 'msiecrawler','muscatferret','netnose-crawler','netpliance','newt', 'noydb','obot','openbot','piranha','psbot','shark','slurp','slurp', 'snap','spider','t-h-u-n-d-e-r-s-t-o-n-e','transmission','vagabond', 'webinator','winhttp','yahoo'); for ($x = 0; $x < count($badWords); $x++) { if (ereg("$badWords[$x]","$user_agent_lc")) { // Got a match - mark as good and move on return 0; } } } return $isHuman; }

« previous php.general (#184897) next »