Re: Need performance help on importer using DB_DataObject

Mike Zornek wrote: >So I've been using DB_DataObject in myproject and am now working on >importing stuff from our old db. I have an ~ 13,000 line CSV file which I am >iterating through. I will first unset the var I use to hold the object, do >some manipulation and then create, set and insert into the db. > >While() { >unset($member); >.... >$member = new DataObjects_Member(); >$member->setMember_id($id); >$member->setMember_first_name($first_name); >$member->setMember_last_name($last_name); >$member->setMember_suffix($suffix); >$member->setMember_birth_date($birthday); >$member->setMember_block_fap($blockfap); >$member->setMember_description($description); >$member->setMember_comments($comments); >$member->setMember_company($company); >$member->setMember_last_modified($last_modified); >$member->setMember_primary_address_id($address1_id); >$member->insert(); >} > >My first concern is how much memory/cpu this uses. True the importer is >never used in production but I'll have to use it a lot over the next few >weeks and as of right now it takes a little over 17 minutes to do it's >thing. I'm on a dual 1.8 GHz G5 Powermac. > >Is there anyway to properly destroy the object? Or maybe reset it, so that I >may use only one instant to insert all my rows--thus cutting down on memory >usage? > >I run the importer from the command line and notice that the php process >uses about 90% of the CPU while importing. The mysqld process jumps around >3-5%. Does this sound right? > >Should I do anything to Mysql to help speed up the import? Turn off indexing >perhaps? > >What do you think is the bottle neck here? Thanks! The bottleneck is probably DataObjects. I don't think it is suited for massive imports like the ones you are describing. You are only trying to do an INSERT so use sql directly. Removing the index might speed it up too but that's not the main bottleneck IMO. If you really need DO, I think there is a setFrom() method that might help. Bertrand Mansion Mamasam