Re: Re: event cache management (no, not code cache) -- W32/Nimda filter

At 23:13 2001-09-21 -0500, Richard Lynch wrote:

You are going to spend a *LOT* of time re-doing the work that went into SQL db that you have eliminated... Re-consider.

I wanted to make the module independant of an SQL service so that it could reach a wider audience by having fewer system requirements.

Under LINUX, so long as your data is less than 4K (?), fopen(...,'a')/fwrite()/fclose() is atomic. So if you are just dumping out IP+timestamp and even some minimal data, you are set.

I'm considering going binary with the data, which means about 12 bytes per record, and makes them fixed length. I can "purge" an item from the cache by overwriting the fields as allzero. Problem is efficiently reading and writing the binary data between an array and the file. Seems I'm going to spend more time massaging the data (either to convert it to/from binary form, or to split it into two arrays), than actually processing it.

If you are *not* using LINUX, your other option is, in fact, to flock() the file. You open it for reading, flock() the handle, fopen()

Yes, I'm well aware of the need to file lock. My concerns lie within having to load the cache db, scan it for a match, deal with updating it as necessary, and whatnot in a modular method -- if I open and close the file between the lookup and update functions, some other process may update the file with the very same IP that I'm dealing with now. So I've really got to open it and keep it open (and locked) until I've completed all my determination and updating. Bugger.

You should also use http://php.net/register_shutdown_function to be sure you never leave a file flock'ed if PHP pukes or something.

Good pointer.

Perhaps even a cron job to release any flocks older than XXX time somehow...

Hmm. This sounds like something PHP should get a new function for: perhaps when a lock is established, an expire could be set by the locking function, and sometime later, when another PHP process attempts to open the file, PHP would realize it is a stale lock (as defined by the registered lock expiration), and remove it... I'm not on PHP-Dev though.

Even *with* LINUX, at some point, you are going to want to shrink the file so it doesn't grow to monster proportions, but you only want to throw away the really old entries...

What I was thinking is that whenever a lookup was performed and resulted in an expired entry, if the expired entry was older than say, 1/2 the expire time, we'd take the opportunity right now to clean up the file.

You see, there's just all sorts of nasty things you need to worry about if you go this route, and you really were better off with SQL where it's all taken care of... Re-consider again. :-)

The design consideration here is that an SQL dependance means anyone who might want to use the module (this is for dealing with CodeRed and W32/Nimda and no doubt the next IIS worm that strikes) must do SQL setup for a new database, etc. oh, and after verifying that the whois protocol is indeed as trivial as I thought is was, I've dealt with that - so at the moment, my script isn't dependant upon any external utilities (obviously, support for updating a firewall would require an external script (fired up from cron), but that isn't required unless you're going to actually enable the firewalling capability, and it still isn't something that is exec'd from the PHP script). --- Please DO NOT carbon me on list replies. I'll get my copy from the list. Sean B. Straw / Professional Software Engineering Post Box 2395 / San Rafael, CA 94912-2395