Edit report at https://bugs.php.net/bug.php?id=69575&edit=1
ID: 69575
User updated by: mark at briley dot com
Reported by: mark at briley dot com
Summary: Mega data - mega problem
-Status: Open
+Status: Closed
Type: Bug
Package: *General Issues
Operating System: Windows 7 Pro
PHP Version: 5.5.24
Block user comment: N
Private report: N
New Comment:
Ugh. I forgot I modified the program to use the fopen, fgets, fwrite, and fclose because of the
problems I was having. I did not keep the original code. :-( Darn it. I am sorting everything as
I read it in now and removing duplicates as each line is read in. :-(
I am going to try to recreate what I had but as it now stands - I can not reproduce the problem
because I completely changed the program and got rid of the array_unique, sort, and any other PHP
specific function that handled arrays. :-(
I'm setting the status to closed because I've gotten past this problem but as I said - I
will see if I can recreate the code and if so - I'll post it.
Previous Comments:
------------------------------------------------------------------------
[2015-05-07 18:46:58] mark at briley dot com
Ok! I just finished modifying and running the same program. For unknown reasons - now PHP
won't go past 2GB. Because of this I changed the program to the following:
<?php
#
# We are going to need a LOT of memory!
# We want 6GB of memory to work on this problem.
#
ini_set( 'memory_limit', '3000M' );
$strings = array();
#
# Make 25,000,000 entries.
#
for ($i = 0; $i < 3000000; $i++) {
$strings[] = uniqid( $i, true );
}
#
# Duplicate all of them so we can see if the duplicates
# sort next to each other.
#
foreach( $strings as $k=>$v ){
$strings[] = $v;
}
#
# Save it so we can edit it via VIM or some other editor.
# All duplicate records should appear right next to each other.
#
$a = implode( "\n", $strings );
file_put_contents( "./out1.dat", $a );
var_dump(strlen($strings[0])); // 67584
var_dump(count($strings)); // 1000
$unique = array_unique($strings);
#
# If the array_unique worked then all of the duplicates
# should be gone. Save it again so we can see if it
# really DID remove all of the non-unique values.
#
$a = implode( "\n", $strings );
file_put_contents( "./out2.dat", $a );
var_dump(count($unique)); // 10
# int(24)
# int(6000000)
# int(3000000)
?>
Note the "3000M" and a reduced set to 3000000 in the FOR loop. When I ran this the above
three numbers were returned. This worked. (So maybe I am hallucinating!) I am going to go back to
the original data and see if it still has problems. (This is just too weird.)
------------------------------------------------------------------------
[2015-05-07 18:19:52] mark at briley dot com
Give me a day. I'll post something as soon as I have it.
------------------------------------------------------------------------
[2015-05-06 21:33:22] cmb@php.net
> The program never got over 3GB but my PHP script never
> complained about the 6000M memory_limit.
That might be regarded as bug, but that's another issue than
array_unique() failing for large arrays, and as such should be
reported as distinct issue.
Anyhow, I wonder how you have been able to run the supplied test
script. All available official Windows builds (x86 as well as x64)
of PHP 5.5.24 don't seem to be able to allocate more than 2G of
memory. Even with a somewhat recent snapshot of an x64 build of
PHP 7.0.0 I have not been able to run the test script (a memory
limit of 6,000,000,000 bytes doesn't suffice, and it seems that
it's not possible to raise the memory limit beyond 8,000,000,000 â
yet another issue).
Can you please give a test script that makes it possible to
reproduce the issue?
------------------------------------------------------------------------
[2015-05-06 18:30:51] mark at briley dot com
Hah! :-) Yeah. The program never got over 3GB but my PHP script never complained about the 6000M
memory_limit. Strange that. :-?
------------------------------------------------------------------------
[2015-05-06 17:24:06] cmb@php.net
Oops! Disregard my last comment; that was nonsense (I had
decreased the number of strings).
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=69575
--
Edit this bug report at https://bugs.php.net/bug.php?id=69575&edit=1