Re: Efficiently parsing a File

Hi Jim, One quick question: When you use this call to the mysql database ("INSERT INTO genotyped (runid, fishid, plateid, assayid, allele1, allele2) VALUES ". join(', ', $dataSet)); There are more than 6 items in the $dataSet array as the while loop parsed the whole file, correct? I just want to make sure I understand correctly. Does that INSERT command knows to start another row in the table after it inserted the first 6 values? Thanks! Tiago On Mar 18, 2014, at 3:38 PM, Jim Lucas <lists@cmsws.com> wrote: > On 03/18/2014 10:57 AM, Tiago Hori wrote: >> Hi Everyone, >> >> I fairly new at this, so please bear with me. :) >> >> I am building this web app for a project I am working at where I need to >> store and process large amounts of data. >> >> The data comes in comma delimited style. There are a bunch of headers that >> I don't need and then the data. These files contain 96 times 96 entries and >> I need to parse each one of those. Right now I have something working that >> takes about 5 minutes to parse the whole file. Any tips on how to make this >> run more efficiently would be greatly appreciated. >> >> Thanks, >> >> Tiago >> >> here is the relevant code: >> >> > > I cleaned up the code a little. Moved a few things around. Removed a few duplicate things and > a part that didn't seem useful. Give this code a try: > > if ( $_FILES['data'] ) { > $dataSet = array(); > $parts = array(); > $c = 0; > $filename = $_FILES['data']['name']; > if ( file_exists($filename) ) { > die ("A file with this name already exists in the database <br />"); > } else { > move_uploaded_file($_FILES['data']['tmp_name'], $filename); > } > $runid = substr($filename, 0, -4); > echo "Uploaded file '$runid' <br />"; > $fh = fopen($filename, 'r') or > die("File does not exist or you lack permission to open it"); > echo '<table>'; > while ( !feof($fh) ) { > $parts = fgetcsv($fh); > if ( $parts[0] == 'ID' ) { > $id = sanitizeStrings($parts[0]); > $assay = sanitizeStrings($parts[1]); > $allele1 = sanitizeStrings($parts[2]); > $allele2 = sanitizeStrings($parts[3]); > $name = sanitizeStrings($parts[4]); > } else if ($t = preg_match("/S\d\d-\D\d\d/", $parts[0]) ) { > $id = sanitizeStrings($parts[0]); > $assay = sanitizeStrings($parts[1]); > $alleles = sanitizeStrings($parts[9]); > $allele1 = $allele2 = 'No Call'; > if ( preg_match("/[ATCG]:[ATCG]/", $alleles) ) { > list($allele1, $allele2) = explode(':', $alleles, 2); > } > $name = sanitizeStrings($parts[4]); > if ($name != 'Blank') { > $dataSet[] = "('{$runid}', '{$name}', > '{$id}', '{$assay}', '{$allele1}', '{$allele2}')"; > } > } > echo <<<_END > <tr> > <th>{$name}</th> > <th>{$id}</th> > <th>{$assay}</th> > <th>{$allele1}</th> > <th>{$allele2}</th> > </tr> > _END; > > } > if ( $dataSet ) { > queryMysql("INSERT INTO genotyped (runid, fishid, plateid, assayid, allele1, > allele2) VALUES ". join(', ', $dataSet)); > } > fclose($fh); > echo '</table>'; > } > > > > -- > Jim Lucas > > http://www.cmsws.com/ > http://www.cmsws.com/examples/