Re: Very Large text file parsing
| From: | Martin Marques | Date: | Thu, 20 Sep 2007 12:54:38 +0000 |
| Subject: | Re: Very Large text file parsing | ||
| References: | 1 2 3 4 | Groups: | php.general |
| Request: | Send a blank email to php-general+get-262291@lists.php.net to get a copy of this message | ||
Paul Scott wrote:
On Thu, 2007-09-20 at 08:03 -0400, Robert Cummings wrote:What's $num and $row for?Post some samples of the data you are parsing and a sample of the code you've written to parse them. If you're parsing 750 megs of data then it's quite likely you could squeeze some performance out of the parse routines themselves.Today's dataset is in a CSV (tab separated) , so I am using fgetcsv, it looks like this (geo data): 936374 Roodepoort Roodepoort Roodeport-Maraisburg -26.1666667 27.8666667 P PPL ZA ZA 06 0 1759 Africa/Johannesburg 2004-05-11 Code: [SNIP] $row = 1; $handle = fopen($csvfile, "r"); while (($data = fgetcsv($handle, 1000, "\t")) !== FALSE) {$num = count($data); $row++;
$insarr = array('userid' => $userid, 'geonameid' => $data[0], 'name' => $data[1], 'asciiname' => $data[2], 'alternatenames' => $data[3], 'latitude' => $data[4], 'longitude' => $data[5], 'featureclass' => $data[6], 'featurecode' => $data[7], 'countrycode' => $data[8], 'cc2' => $data[9], 'admin1code' => $data[10], 'admin2code' => $data[11], 'population' => $data[12], 'elevation' => $data[13], 'gtopo30' => $data[14], 'timezoneid' => $data[15], 'moddate' => $data[16]
);
$this->objDbGeo->insertRecord($insarr);