Copying large volumes of data to a DB
| From: | Alan Lord | Date: | Mon, 04 Jul 2005 21:45:04 +0000 |
| Subject: | Copying large volumes of data to a DB | ||
| Groups: | php.db | ||
| Request: | Send a blank email to php-db+get-39872@lists.php.net to get a copy of this message | ||
Hi All,
I'm starting to build an app to manage my eBay sales. Now one of the things I need to do is to write the Category information into a local DB.
Once the initial download is done I'm O.K. about updating the content but my questions relates more to this:
What is the best way to manage the Mbytes of data which will be coming down a relatively slow line (DSL) and then writing it into my DB.
Should I have some sort of buffer where I read n bytes or records and then write those to the DB and repeat until the process is finished, or, do I wait until the lot has downloaded then write it to my DB in one big gulp?
I've done plenty or read/writinig to/from DBs but it has always single records in the past and I'm just wondering about the "best" way to handle this?
Thanks in advance
Alan