Bug #69575 [Opn]: Mega data - mega problem

Edit report at https://bugs.php.net/bug.php?id=69575&edit=1 ID: 69575 User updated by: mark at briley dot com Reported by: mark at briley dot com Summary: Mega data - mega problem Status: Open Type: Bug Package: *General Issues Operating System: Windows 7 Pro PHP Version: 5.5.24 Block user comment: N Private report: N New Comment: Ok! I just finished modifying and running the same program. For unknown reasons - now PHP won't go past 2GB. Because of this I changed the program to the following: <?php # # We are going to need a LOT of memory! # We want 6GB of memory to work on this problem. # ini_set( 'memory_limit', '3000M' ); $strings = array(); # # Make 25,000,000 entries. # for ($i = 0; $i < 3000000; $i++) { $strings[] = uniqid( $i, true ); } # # Duplicate all of them so we can see if the duplicates # sort next to each other. # foreach( $strings as $k=>$v ){ $strings[] = $v; } # # Save it so we can edit it via VIM or some other editor. # All duplicate records should appear right next to each other. # $a = implode( "\n", $strings ); file_put_contents( "./out1.dat", $a ); var_dump(strlen($strings[0])); // 67584 var_dump(count($strings)); // 1000 $unique = array_unique($strings); # # If the array_unique worked then all of the duplicates # should be gone. Save it again so we can see if it # really DID remove all of the non-unique values. # $a = implode( "\n", $strings ); file_put_contents( "./out2.dat", $a ); var_dump(count($unique)); // 10 # int(24) # int(6000000) # int(3000000) ?> Note the "3000M" and a reduced set to 3000000 in the FOR loop. When I ran this the above three numbers were returned. This worked. (So maybe I am hallucinating!) I am going to go back to the original data and see if it still has problems. (This is just too weird.) Previous Comments: ------------------------------------------------------------------------ [2015-05-07 18:19:52] mark at briley dot com Give me a day. I'll post something as soon as I have it. ------------------------------------------------------------------------ [2015-05-06 21:33:22] cmb@php.net > The program never got over 3GB but my PHP script never > complained about the 6000M memory_limit. That might be regarded as bug, but that's another issue than array_unique() failing for large arrays, and as such should be reported as distinct issue. Anyhow, I wonder how you have been able to run the supplied test script. All available official Windows builds (x86 as well as x64) of PHP 5.5.24 don't seem to be able to allocate more than 2G of memory. Even with a somewhat recent snapshot of an x64 build of PHP 7.0.0 I have not been able to run the test script (a memory limit of 6,000,000,000 bytes doesn't suffice, and it seems that it's not possible to raise the memory limit beyond 8,000,000,000 – yet another issue). Can you please give a test script that makes it possible to reproduce the issue? ------------------------------------------------------------------------ [2015-05-06 18:30:51] mark at briley dot com Hah! :-) Yeah. The program never got over 3GB but my PHP script never complained about the 6000M memory_limit. Strange that. :-? ------------------------------------------------------------------------ [2015-05-06 17:24:06] cmb@php.net Oops! Disregard my last comment; that was nonsense (I had decreased the number of strings). ------------------------------------------------------------------------ [2015-05-06 17:15:20] cmb@php.net Gee! Setting the memory_limit to '6000000000' instead of '6000M' did work (so disregard my former comment about a general 2G memory limit), and the test script gave the expected results. ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=69575 -- Edit this bug report at https://bugs.php.net/bug.php?id=69575&edit=1