Bug #69575 [Csd->ReO]: Mega data - mega problem

Edit report at https://bugs.php.net/bug.php?id=69575&edit=1 ID: 69575 Updated by: ab@php.net Reported by: mark at briley dot com Summary: Mega data - mega problem -Status: Closed +Status: Re-Opened Type: Bug Package: *General Issues Operating System: Windows 7 Pro PHP Version: 5.5.24 Block user comment: N Private report: N New Comment: For the huge memory_limit, please use some master x64 snapshot. PHP5 on windows can't physically take this. Thanks. Previous Comments: ------------------------------------------------------------------------ [2015-05-07 19:10:54] mark at briley dot com Ugh. I forgot I modified the program to use the fopen, fgets, fwrite, and fclose because of the problems I was having. I did not keep the original code. :-( Darn it. I am sorting everything as I read it in now and removing duplicates as each line is read in. :-( I am going to try to recreate what I had but as it now stands - I can not reproduce the problem because I completely changed the program and got rid of the array_unique, sort, and any other PHP specific function that handled arrays. :-( I'm setting the status to closed because I've gotten past this problem but as I said - I will see if I can recreate the code and if so - I'll post it. ------------------------------------------------------------------------ [2015-05-07 18:46:58] mark at briley dot com Ok! I just finished modifying and running the same program. For unknown reasons - now PHP won't go past 2GB. Because of this I changed the program to the following: <?php # # We are going to need a LOT of memory! # We want 6GB of memory to work on this problem. # ini_set( 'memory_limit', '3000M' ); $strings = array(); # # Make 25,000,000 entries. # for ($i = 0; $i < 3000000; $i++) { $strings[] = uniqid( $i, true ); } # # Duplicate all of them so we can see if the duplicates # sort next to each other. # foreach( $strings as $k=>$v ){ $strings[] = $v; } # # Save it so we can edit it via VIM or some other editor. # All duplicate records should appear right next to each other. # $a = implode( "\n", $strings ); file_put_contents( "./out1.dat", $a ); var_dump(strlen($strings[0])); // 67584 var_dump(count($strings)); // 1000 $unique = array_unique($strings); # # If the array_unique worked then all of the duplicates # should be gone. Save it again so we can see if it # really DID remove all of the non-unique values. # $a = implode( "\n", $strings ); file_put_contents( "./out2.dat", $a ); var_dump(count($unique)); // 10 # int(24) # int(6000000) # int(3000000) ?> Note the "3000M" and a reduced set to 3000000 in the FOR loop. When I ran this the above three numbers were returned. This worked. (So maybe I am hallucinating!) I am going to go back to the original data and see if it still has problems. (This is just too weird.) ------------------------------------------------------------------------ [2015-05-07 18:19:52] mark at briley dot com Give me a day. I'll post something as soon as I have it. ------------------------------------------------------------------------ [2015-05-06 21:33:22] cmb@php.net > The program never got over 3GB but my PHP script never > complained about the 6000M memory_limit. That might be regarded as bug, but that's another issue than array_unique() failing for large arrays, and as such should be reported as distinct issue. Anyhow, I wonder how you have been able to run the supplied test script. All available official Windows builds (x86 as well as x64) of PHP 5.5.24 don't seem to be able to allocate more than 2G of memory. Even with a somewhat recent snapshot of an x64 build of PHP 7.0.0 I have not been able to run the test script (a memory limit of 6,000,000,000 bytes doesn't suffice, and it seems that it's not possible to raise the memory limit beyond 8,000,000,000 – yet another issue). Can you please give a test script that makes it possible to reproduce the issue? ------------------------------------------------------------------------ [2015-05-06 18:30:51] mark at briley dot com Hah! :-) Yeah. The program never got over 3GB but my PHP script never complained about the 6000M memory_limit. Strange that. :-? ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=69575 -- Edit this bug report at https://bugs.php.net/bug.php?id=69575&edit=1