Re: Re: Efficiently parsing a File

On 03/18/2014 02:29 PM, Tiago Hori wrote:

Thanks All. I am looking at all your suggestions. The file has 9216 entries with 10 columns, but I only need 2 of them really.

When you say need, does that mean that you are only inserting those two rows into the database, or on the other side of the if..else ?

The crucial one, is the one that I have to split with explode using the “: “ separator.

crucial in what way? Is this the side that has the two rows or the remaining 9214 rows?

The files are about 1MB, that’s why I reached out, I figured it shouldn’t take that long.

Correct, it shouldn't. Sounds like something (more then likely the DB inserts) that are taking so long. Depending on how your table is built and how much data you already have in there, re-indexing the table could be what is slowing you down.

Thanks again! T. On Mar 18, 2014, at 5:55 PM, Christoph Becker <cmbecker69@gmx.de> wrote:

Tiago Hori wrote:

I fairly new at this, so please bear with me. :) I am building this web app for a project I am working at where I need to store and process large amounts of data. The data comes in comma delimited style. There are a bunch of headers that I don't need and then the data. These files contain 96 times 96 entries

Do you mean 96 records (rows) with 96 columns each, or do you mean 9216 records? What's the size of the file?

                                                                        and

I need to parse each one of those. Right now I have something working that takes about 5 minutes to parse the whole file.

I doubt that *parsing* the file takes about 5 minutes.

                                                Any tips on how to make this

run more efficiently would be greatly appreciated.

Measure first where you're script spends time, before you optimize in the "wrong" place. A profiler such as Xdebug's or xhprof should come in handy. If you don't have a profiler available, I suggest you replace queryMysql($query) with true, and check how long it takes to process the file. If that is much faster than before (and I presume so), you'll have to optimize the insertion of the values into the database. -- Christoph M. Becker

-- Jim Lucas http://www.cmsws.com/ http://www.cmsws.com/examples/