Re: Individual bulk e-mails - performance question (was "skinning a cat") :-)

On Wed, 2008-09-03 at 00:55 -0500, Micah Gersten wrote: > Robert Cummings wrote: > > On Mon, 2008-09-01 at 14:34 +0200, Merlin Morgenstern wrote: > > > >> Per Jessen schrieb: > >> > >>> Jochem Maas wrote: > >>> > >>> > >>>>> lockfile=/var/lock/<xxxx>/file > >>>>> # clear out a lock older than 60mins > >>>>> find $lockfile -cmin +60 | xargs rm > >>>>> test ! -f $lockfile && ( > >>>>> touch $lockfile > >>>>> <run some php> > >>>>> rm -f $lockfile > >>>>> ) > >>>>> > >>>> wouldn't creating a dir be better here? (with regard to atomicity) > >>>> > >>>> > >>> I haven't thought it through, but I don't think it would change > >>> anything. > >>> > >>> > >>> /Per Jessen, Zürich > >>> > >>> > >> Hello everybody, > >> > >> thank you for the huge amount of replies on my question. I believe I > >> quite lost a bit track on how to solve the problem hands on. To > >> summerize, there are aparently two ways of skinning the cat :-) > >> > >> 1. Run a deamon > >> I admit I have never wrote one, and that seems therefore to be the > >> harder option for me. Well... perhaps not. > >> I assume it is simply creating a shell script like Jochem wrote, however > >> I guess it would just take longer for me to get it running as I have > >> never done that before. > >> > >> 2. Create a PHP file that holds some lock code which is triggered by > >> CRON every 5 minutes. Taking the example from Robert. > >> > >> 3. Following idea which I am now actually implementing :-) > >> > >> - Add all emails into a mysql db with one insert command. > >> - Run a php script by cron every 5 min that does the following: > >> - update table, set session_id = x on all entries without session id > >> - get all entries with that session ID > >> - send email to those with that session ID > >> - remove all entries with that session ID > >> > >> Looks like easies sollution on how to skimm the cat. DB might not be the > >> most performing sollution, but sufficient considering the hard ware power. > >> > > > > Actually 3 was what I suggested with a lock mechanism to ensure that if > > your script run by cron takes longer than 5 minutes that you don't have > > two scritps stepping on each others' toes. > > > > Cheers, > > Rob. > > > He solves that by adding the session ID at the beginning of the script > and only operating on those entries. The scripts might be running > simultaneously, but will not step on each other's toes. Ahh, I didn't notice he added extra details. Oh well... unless he locks the table then he's still got race conditions to face. But i guess they're unlikely to be an issue given that there's 5 minutes between one run and the next. The other problem is if the system were to grind to a halt because cron2 fires before cron1 finishes, and cron3 fires before cron1 and cron2 finish, and cron4... well you get the picture. With a single cron script in execution at any given time you can be sure that not everyone is trying to squeeze throught he door at the same time. Cheers, Rob. -- http://www.interjinn.com Application and Templating Framework for PHP