Re: Re: Efficiently parsing a File

On 3/18/2014 7:19 PM, Tiago Hori wrote:

Jim and Christoph, Thanks!

I am looking at all your suggestions. The file has 9216 entries with 10 columns, but I only need 2 of them really. The crucial one, is the one that I have to split with explode using the “: “ separator. The files are about 1MB, that’s why I reached out, I figured it shouldn’t take that long.

I still presume the performance problem is due to the many separate insert statements (reading such files with fgets() isn't the problem). There is an article regarding the speed of insertions in MySQL databases, with several optimization suggestions: <https://dev.mysql.com/doc/refman/5.7/en/insert-speed.html> HTH

So, I could use Jim suggestion, but maybe not add the whole 9000 entries of time, correct? Would it be a good solution to create separate arrays with every 500 rows or just create one big array like Jim suggested and then break the insertion into iterations of 100 rows? Thanks! T, write out a temporary flat file the way you need it to look in the database delimited however you need it delimited. Then use "LOAD DATA INFILE" to populate the table. LOAD DATA INFILE is very fast and it's very efficient. It'll load a 1MB file in a fraction of a second.

delete the flat file when you're done. --Curtis