Bug #69575 [Fbk->Opn]: Mega data - mega problem

Edit report at https://bugs.php.net/bug.php?id=69575&edit=1 ID: 69575 User updated by: mark at briley dot com Reported by: mark at briley dot com Summary: Mega data - mega problem -Status: Feedback +Status: Open Type: Bug Package: *General Issues Operating System: Windows 7 Pro PHP Version: 5.5.24 Block user comment: N Private report: N New Comment: Hah! :-) Yeah. The program never got over 3GB but my PHP script never complained about the 6000M memory_limit. Strange that. :-? Previous Comments: ------------------------------------------------------------------------ [2015-05-06 17:24:06] cmb@php.net Oops! Disregard my last comment; that was nonsense (I had decreased the number of strings). ------------------------------------------------------------------------ [2015-05-06 17:15:20] cmb@php.net Gee! Setting the memory_limit to '6000000000' instead of '6000M' did work (so disregard my former comment about a general 2G memory limit), and the test script gave the expected results. ------------------------------------------------------------------------ [2015-05-06 17:06:23] cmb@php.net Hmm, I can't run the given test script, because there is a 2 GB memory limit in 5.5.24 (x64 as well as experimental x64 build) on Windows (besides a typo in the script: $strings[$v] should be $strings[$k]). If I'm not mistaken this memory limit applies to all current PHP versions. This might have actually been the cause of your script misbehaving. | we are taking thousands of XML files, putting them together, | converting them to SQL, ensuring they are going into the | database in the right order (sorting), and then splitting the | SQL file up into separate files. It might be a workaround to split the big SQL first, then sorting each file separately, and then doing a merge sort[1] on the files. [1] <http://en.wikipedia.org/wiki/Merge_sort> ------------------------------------------------------------------------ [2015-05-06 15:35:13] mark at briley dot com As stated in the problem, small data sets work just fine. Try changing your data set size by doing the following: # # We are going to need a LOT of memory! # We want 6GB of memory to work on this problem. # ini_set( 'memory_limit', '6000M' ); $strings = array(); # # Make 25,000,000 entries. # for ($i = 0; $i < 25000000; $i++) { $strings[] = uniqid( $i, true ); } # # Duplicate all of them so we can see if the duplicates # sort next to each other. # foreach( $strings as $k=>$v ){ $strings[] = $strings[$v]; } # # Save it so we can edit it via VIM or some other editor. # All duplicate records should appear right next to each other. # file_put_contents( "./out1.dat", $strings ); var_dump(strlen($strings[0])); // 67584 var_dump(count($strings)); // 1000 $unique = array_unique($strings); # # If the array_unique worked then all of the duplicates # should be gone. Save it again so we can see if it # really DID remove all of the non-unique values. # file_put_contents( "./out2.dat", $strings ); var_dump(count($unique)); // 10 NOW see if it works. I found this same error years ago in the PERL code. The problem was that PERL was using shorts instead of longs in its sorting routines. This meant that it worked up to 65,536 records but if you exceeded that many records it borked and returned only SOME of the array sorted. I suspect this is that same problem showing up again. Your next question to me should be "Why the heck are you sorting so many records in memory?" The answer is - we are taking thousands of XML files, putting them together, converting them to SQL, ensuring they are going into the database in the right order (sorting), and then splitting the SQL file up into separate files. We are having to do this because we are not going for our own server and instead are using one of the cheaper setups. Unfortunately, the cheaper setup won't let us upload all of the one large file at one time. No file can be larger than 10MB and due to problems when we were really close to the 10MB limit - I'm making sure the files are no larger than 5MB each. What can I say? We are in the initial stages of setting everything up and until it is shown that the whole thing will work - no more money is going to be used for this project. (Other than my salary.) Once everything is up and running we will probably get a dedicated server. Ok - next question - "Why don't we just upload the XML files?" The company we are going through is using an older version of phpMyAdmin and only CSV and SQL files can be uploaded. Thus - we have to convert the XML files over to SQL files. We can not use CSV because of the size of some of these files. CSV will croak if the size of a record is greater than....drum roll please.....65,536 characters. Thank you 8bit computers for making companies NOT switch to 32bit technologies. It is like that old joke. Why are roads the width they are? The joke drags on until you get to the real reason which is : "The romans built the original roads and these roads HAD to be the same width as a chariot which was pulled by two horses." Thus proving two horse's rears take precedent over what something actually should be and also proving once a standard is created it stays around forever. :-) (ie: shorts over longs because that's how it has ALWAYS been done) ------------------------------------------------------------------------ [2015-05-05 17:32:07] cmb@php.net The following test script gives the expected results (PHP 5.5.24, Win7): $strings = array(); for ($i = 0; $i < 1000; $i++) { $strings[] = str_repeat($i % 10, 66 * 1024); } var_dump(strlen($strings[0])); // 67584 var_dump(count($strings)); // 1000 $unique = array_unique($strings); var_dump(count($unique)); // 10 To be able to reproduce the issue, we would need respective sample data. Can you make these available for download somewhere? ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=69575 -- Edit this bug report at https://bugs.php.net/bug.php?id=69575&edit=1