Re: Working with large datasets

On 2011-10-10, at 11:30 AM, Jason Pruim <lists@pruimphotography.com> wrote: > Hey everyone, > > > I am working with a database that has close to 8 million records in it and it will be growing. > I have a state field in the data, and I am attempting to test some query's on it, all but 2 > records right now have the same state. > > My test info won't get pulled up... I believe it keeps timing out the connection. > > Is there any advice for working with large datasets? I'm wanting this to be able to load > quickly. > > Thanks in advance! > > > Jason Pruim > lists@pruimphotography.com > > > > > -- > PHP Database Mailing List (http://www.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php > Assuming mysql, what is the my.conf set for? Check that you are using the large dataset one. By default it's usually a small one. That will give you more memory and sort spaces work with the data. We routinely handle 8-10mm records and it's not tough. The tricks are 1: ensure enough sort space 2: ensure enough memory for large sets 3: ensure about php memory for results 4: try to add additional filters to reduce the data sets. A cardinality of two on a status will always return tons of records and you want to reduce that, maybe with a date range Bastien Koert 905-904-0334