Edit report at https://bugs.php.net/bug.php?id=69575&edit=1
ID: 69575
Updated by: php-bugs@lists.php.net
Reported by: mark at briley dot com
Summary: Mega data - mega problem
-Status: Feedback
+Status: No Feedback
Type: Bug
Package: *General Issues
Operating System: Windows 7 Pro
PHP Version: 5.5.24
Private report: N
New Comment:
No feedback was provided. The bug is being suspended because
we assume that you are no longer experiencing the problem.
If this is not the case and you are able to provide the
information that was requested earlier, please do so and
change the status of the bug back to "Re-Opened". Thank you.
Previous Comments:
------------------------------------------------------------------------
[2015-06-02 17:59:18] ab@php.net
@mark, if you're willing to test this, please do. Otherwise the ticket will be auto closed if
there's no feedback.
Thanks.
------------------------------------------------------------------------
[2015-05-08 05:55:48] ab@php.net
For the huge memory_limit, please use some master x64 snapshot. PHP5 on windows can't
physically take this.
Thanks.
------------------------------------------------------------------------
[2015-05-07 19:10:54] mark at briley dot com
Ugh. I forgot I modified the program to use the fopen, fgets, fwrite, and fclose because of the
problems I was having. I did not keep the original code. :-( Darn it. I am sorting everything as
I read it in now and removing duplicates as each line is read in. :-(
I am going to try to recreate what I had but as it now stands - I can not reproduce the problem
because I completely changed the program and got rid of the array_unique, sort, and any other PHP
specific function that handled arrays. :-(
I'm setting the status to closed because I've gotten past this problem but as I said - I
will see if I can recreate the code and if so - I'll post it.
------------------------------------------------------------------------
[2015-05-07 18:46:58] mark at briley dot com
Ok! I just finished modifying and running the same program. For unknown reasons - now PHP
won't go past 2GB. Because of this I changed the program to the following:
<?php
#
# We are going to need a LOT of memory!
# We want 6GB of memory to work on this problem.
#
ini_set( 'memory_limit', '3000M' );
$strings = array();
#
# Make 25,000,000 entries.
#
for ($i = 0; $i < 3000000; $i++) {
$strings[] = uniqid( $i, true );
}
#
# Duplicate all of them so we can see if the duplicates
# sort next to each other.
#
foreach( $strings as $k=>$v ){
$strings[] = $v;
}
#
# Save it so we can edit it via VIM or some other editor.
# All duplicate records should appear right next to each other.
#
$a = implode( "\n", $strings );
file_put_contents( "./out1.dat", $a );
var_dump(strlen($strings[0])); // 67584
var_dump(count($strings)); // 1000
$unique = array_unique($strings);
#
# If the array_unique worked then all of the duplicates
# should be gone. Save it again so we can see if it
# really DID remove all of the non-unique values.
#
$a = implode( "\n", $strings );
file_put_contents( "./out2.dat", $a );
var_dump(count($unique)); // 10
# int(24)
# int(6000000)
# int(3000000)
?>
Note the "3000M" and a reduced set to 3000000 in the FOR loop. When I ran this the above
three numbers were returned. This worked. (So maybe I am hallucinating!) I am going to go back to
the original data and see if it still has problems. (This is just too weird.)
------------------------------------------------------------------------
[2015-05-07 18:19:52] mark at briley dot com
Give me a day. I'll post something as soon as I have it.
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=69575
--
Edit this bug report at https://bugs.php.net/bug.php?id=69575&edit=1