Re: Working with large datasets

Jason Pruim lists@pruimphotography.com On Oct 10, 2011, at 5:27 PM, Thompson, Jimi wrote: > I really think that you should try running it from the command line and see what the issues > are. Get both Apache and php out of the way. I've seen some PHP scripts use up all the file > handles (OS limit) even on a 64 bit server when they start doing complex things with data sets. > > If it works ok without PHP/Apache then you can start lookig at PHP and APache. > > ISOLATE the issue not complicate it.... > > My 2 cents, Hi Jimi, I've done it from the command line a few times, first time I run a simple: SELECT * FROM Main WHERE state="test"; it takes: 2 rows in set (1 min 44.20 sec) after that initial run it's pretty responsive, usually around 0.01 seconds. If I do a select based on the new-york state, which is all of my almost 9 million records after the initial run, it returns it all VERY quickly: 25000 rows in set (0.02 sec) So commandline is running fine after the initial... I'm leaning towards either a problem with web server, PHP setup, or my PHP code... I've used the code many times before but never on such a large dataset... When I pull the pagination out completely.... It's pretty much the same result... fetching the info for new-york works just fine but not "test" One thing I am noticing right now though is the fact that when I switch over to using the test state.... Right now it's not displaying anything... Not even able to view the source... Okay... Enough rambling right now... Need to do some more checking before I can come to a conclusion :)