Re: log file analysis

on 7/25/00 1:20 PM, R. P. Channing Rodgers, M.D. at rodgers@nlm.nih.gov wrote: >> From: Kris Dahl <kris@toolhouse.com> >> Subject: [PHP] log file analysis >> >> I am wondering what you all are using to do log file analysis, etc. >> >> We are running into some issues with PHP in that a lot of log file analysis >> tools are not very 'php/cgi/dynamic content' friendly > [...] >> The closest I can find is Webalizer--which is pretty cool but still has some >> catch up to play before they are on the level with Web Trends, etc. > > Another popular open source package is analog: > > http://analog.technomancer.com/ Analog I have used before. It is pretty good--but one key thing that we need for these reports is the ability for them to be easy to read, i.e. not just for us geeks. > for which there is a higher-level add-on perl package with nice graphics > called Report Magic: > > http://www.wadsack-allen.com/digitalgroup/reportmagic/ This helps bridge the gap--it feels more like Webtrends. I would definitely consider further developing this application. > But your point about PHP dynamic content being harder to get good > numbers on is well taken, and I don't know if these packages address > that issue. It's an important problem to solve... Well, its a very tough problem. Coming up with a 'universal' log file parser is pretty tough. Consider http://www.mysite.com/download.php?file_id=15 and http://www.myssite.com/register.php?email=krislists@toolhouse.com The first script would stream a file out of a BLOB in a database. We'd like to keep track of 'file' that was downloaded. But you may want register.php to be treated as a 'single' file so that we can know how many times inclusive that someone has registered, not how many times *i* registered. Perhaps that is where logging will be headed--or needs to. You may have to define the behavior of each file on the server and how to deal with the query string. Of course that opens a big can of worms--how do you implement even that? To make it easy enough for people to use/get adopted you may have to use a 'do as I mean' language, which I personally despise (at least how most are implemented. Then further yet, do you say screw trying to extract info from an Apache or CLF log file altogether? For 'real-time' logging have implemented 'checkpoints' that we are able to track users--i.e we have x number of users that have items in their shopping carts right now, we have x number of users that are confirming their shipping info, etc. Should I step it up to a higher level and log everything? I am leaning towards logging to a MySQL database using that mod_mysql_log or whatever and then performing simple SQL queries. That would make customizing reports pretty easy, I think. Ugh. No rest for the wicked. -k