Re: Issue with the huge import script

On Aug 15, 2009, at 9:15 PM, Devendra Jadhav <devendra.in@gmail.com> wrote:

Hi, I have to import data from one database to another, I have to import around 100000(1Lac) records. First I need to check if the record is already imported or not and import only those records which are not imported. Here is my logic $already_imported = get_already_imported_records(); format of the $already_imported is $already_imported[someid] = 'imported'; Now i take all records from another db and iterating through it. if (!key_exists($already_imported[$new_id])){

       import_function($new_id)

}else{

       echo 'allready imported'.$already_imported[$new_id];

} Now my script is importing same records for more than one time. I am not able to get through this issue Is it because of the size of the records or something else...? Please suggest me some solution which is faster, safe and easy to code :D Thanks in advance --Devendra Jadhav

What are the databases? Both mysql? Or different systems? How do you define whether the record exists? Are you running into timeout issues? If so, one trick is to write the page to process 100 or so records at a time and then use JavaScript to reload the page to avoid the timeout issues. Bastien Sent from my iPod