Re: Add New Records Only!

Ave, > A better solution would be to add a column in the MySQL table, maybe call it > "processed" with a default value of 0, and update this value to 1 with each > row inserted. Then you are only querying records where processed=0. > Of course this will not work if you cannot modify the MySQL table. I have to tell you that this is probably the most efficient & feasible method suggested yet! Let me take a look at my structure & application and see how efficiently this can be implemented. I did think about the cron job midnight update initially - and while it looked like a possible method, my only problem was that occasionally a manual update would be required, and I would need a program in place that is able to handle that. If it were solely an automatic midnight update, I would have certainly gone the cron way with a date limitation. Thanks! On 3/23/07 2:17 PM, "Brad Fuller" <bfuller@cpacampaigns.com> wrote: > Rahul wrote: >> Ave, >> >> It's definitely not live data, so that is not a problem at all. But I'm >> not >> sure I understand your method very well. >> >> I do understand getting data from both the existing DBF and the multiple >> mySQL tables into a temporary mySQL table. But if I do go ahead and do >> that, >> I guess I could write a 'delete-duplicates' kind of code that deletes all >> rows in that temporary table which are duplicates, and then add the >> leftover >> into the DBF. >> >> Not sure how this sounds, or how close this is to what you were saying. >> And >> not even sure how to implement this. > > > Do you need to update this more than once a day? Is there a date field in > all the tables? > > If you can do it daily, then query for records from the previous day and run > it once daily at midnight via cron job. > > If it has to be done more often than once a day there are other solutions. > > One thing I thought is to store the name of the table it came from along > with the primary key in that table. These 2 fields combined will be your > unique identifier. Store this in your DBF and check for existence before > you insert. > > With that solution in mind however you're still querying for the entire set > of data which is very inefficient. > > A better solution would be to add a column in the MySQL table, maybe call it > "processed" with a default value of 0, and update this value to 1 with each > row inserted. Then you are only querying records where processed=0. > Of course this will not work if you cannot modify the MySQL table. > > Best of luck, > > Brad