Re: What is the best way to process live data?

On Sat, Aug 29, 2009 at 3:31 PM, Paul Halliday<paul.halliday@gmail.com> wrote: > For those of you that remember (not likely but anyway) I am working on > some code that splits CLF records and feeds them into a database. > > What I need to do now is automate it. > > So what I have is a program (urlsnarf) that redirects its output > (simple "cmd > file.txt") to a file. The script currently processes > this output line by line and does the SQL foo - i.e. script.php > /the/file.txt > > Should I: > > 1) do away with redirecting the output from urlsnarf to a file and > have the script run the process and have some kind of handle on it; > foreach kinda thing? > 2) keep outputting the info to a persistent file, reading any new > appends and periodically cull this file. Seems like a waste of I/O if > #1 is possible. > 3) other options? > > Any help/push in the right direction is appreciated. > > Thanks. > I'd do away with the text file and snarf to my DB to begin with.