RE: [PHP] Parsing access_log entries

I think this is probably a job that could be better solved by perl since you can also have apache log to a pipe (i.e. to a script/program). This will also allow you to do some preprocessing before you do your database insert, which can prove valuable for larger sites (1mil+ inserts a day can hurt :) I also don't believe (i could be wrong here) that apache will block on logging, so if the database is for some reason down or slow, then your websites performance won't suffer. If you're concerned about losing information in that case, implement store&forward in your script/program and all should be well :) Note that you could also try implementing this with php compiled as a seperate executable (not a webserver module). I myself am not so good with regexp, but for parsing logs you probably don't want something that slow. You should use split() as much as possible and regexp sparingly if at all. Think in chunks. Hope this helps. -jm -----Original Message----- From: jeremy brand [mailto:jeremy@nirvani.net] Sent: Thursday, December 07, 2000 7:04 AM To: John Biesnecker Cc: php-general@lists.php.net Subject: Re: [PHP] Parsing access_log entries See: http://www.nirvani.net/software/j-sessions-1.0.0-pre1 J-sessions includes support for custom logging. There is an SQL schema and my_logme() function that will allow you to log in real time (to the DB). With this, you would not even have to create apache logs, and you would be better off not logging twice anyway... As you can see, you could change the schema to do whatever you want. They, put my_logme() either in each file that you want to log, or put it in the php.ini file as to be included in each page. You might also want to take out some of the KEYs if you are going to use it on a high traffic site. Here is the code: /** This is the schema for the log table used in the my_logme() function. CREATE TABLE logs ( id int(9) unsigned DEFAULT '0' NOT NULL auto_increment, request_uri varchar(50) DEFAULT '' NOT NULL, host_name varchar(25) DEFAULT '' NOT NULL, remote_addr varchar(15) DEFAULT '0.0.0.0' NOT NULL, http_referer blob NOT NULL, user_agent varchar(75) DEFAULT '' NOT NULL, ts int(9) unsigned DEFAULT '0' NOT NULL, PRIMARY KEY (id), KEY request_uri (request_uri), KEY user_agent (user_agent), KEY ts (ts) ); **/ function my_logme() { global $HTTP_HOST, $REQUEST_URI, $HTTP_REFERER, $HTTP_USER_AGENT, $REMOTE_ADDR; /** Don't log if it is us **/ if ($REMOTE_ADDR == 'xxx.xxx.xxx.xxx' || $REMOTE_ADDR == 'xxx.xxx.xxx.xxx' || ereg('xxx\.xxx\.xxx',$REMOTE_ADDR)) return TRUE; $ts = time(); $q = "INSERT into logs set request_uri='$REQUEST_URI', host_name='$HTTP_HOST', remote_addr='$REMOTE_ADDR', http_referer='$HTTP_REFERER', user_agent='$HTTP_USER_AGENT', ts=$ts"; mysql_query($q); return TRUE; Jeremy Brand :: Sr. Software Engineer :: 408-245-9058 :: jeremy@nirvani.net http://www.JeremyBrand.com/Jeremy/Brand/Jeremy_Brand.html for more Get your own Free, Private email at http://www.smackdown.com/ - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - At least for the people who send me mail about a new language that they're designing, the general advice is: do it to learn about how to write a compiler. --Dennis Ritchie On Thu, 7 Dec 2000, John Biesnecker wrote: > Date: Thu, 7 Dec 2000 17:54:15 -0500 > From: John Biesnecker <amraam@ao.net> > To: Joe Stump <joe@earth.care2.com> > Cc: php-general@lists.php.net > Subject: Re: [PHP] Parsing access_log entries > > The problem with that method is that there are spaces within each line that > aren't field deliminators. That's why I was thinking regular expressions... > > Thanks, though. > > /s/ John > > ----- Original Message ----- > From: "Joe Stump" <joe@earth.care2.com> > To: "John Biesnecker" <amraam@ao.net> > Cc: <php-general@lists.php.net> > Sent: Thursday, December 07, 2000 5:45 PM > Subject: Re: [PHP] Parsing access_log entries > > > > Real time eh? Well I would recommend this: > > > > <? > > > > $fp = fopen('/dev/stdin','r'); > > while(!feof($fp)) > > { > > $line = trim(fgets($fp,4096)); > > $array_of_line = explode(' ',$line); > > // parse based on that. > > } > > > > ?> > > > > Or in your main include file (we all have one!) put a function in the logs > the > > info you want to a DB. > > > > --Joe > > > > > > On Thu, Dec 07, 2000 at 05:41:31PM -0500, John Biesnecker wrote: > > > Hello all.... > > > > > > I'm trying to parse my Apache access_log files in real time. I'm > looking > > > for a function or something to split a line in the access_log into an > array > > > that i could then parse. I was thinking a regular expression would do > the > > > trick, but my skills in that area are poor to none. > > > > > > Thanks for the help. > > > > > > /s/ John > > > > > > > > > -- > > > PHP General Mailing List (http://www.php.net/) > > > To unsubscribe, e-mail: php-general-unsubscribe@lists.php.net > > > For additional commands, e-mail: php-general-help@lists.php.net > > > To contact the list administrators, e-mail: php-list-admin@lists.php.net > > > > --- > > Joe Stump > > PHP Programmer > > www.Care2.com > > > > > -- > PHP General Mailing List (http://www.php.net/) > To unsubscribe, e-mail: php-general-unsubscribe@lists.php.net > For additional commands, e-mail: php-general-help@lists.php.net > To contact the list administrators, e-mail: php-list-admin@lists.php.net > > -- PHP General Mailing List (http://www.php.net/) To unsubscribe, e-mail: php-general-unsubscribe@lists.php.net For additional commands, e-mail: php-general-help@lists.php.net To contact the list administrators, e-mail: php-list-admin@lists.php.net