Edit report at https://bugs.php.net/bug.php?id=69575&edit=1
ID: 69575
Updated by: cmb@php.net
Reported by: mark at briley dot com
Summary: Mega data - mega problem
-Status: Open
+Status: Feedback
Type: Bug
Package: *General Issues
Operating System: Windows 7 Pro
PHP Version: 5.5.24
Block user comment: N
Private report: N
New Comment:
The following test script gives the expected results (PHP 5.5.24,
Win7):
$strings = array();
for ($i = 0; $i < 1000; $i++) {
$strings[] = str_repeat($i % 10, 66 * 1024);
}
var_dump(strlen($strings[0])); // 67584
var_dump(count($strings)); // 1000
$unique = array_unique($strings);
var_dump(count($unique)); // 10
To be able to reproduce the issue, we would need respective sample
data. Can you make these available for download somewhere?
Previous Comments:
------------------------------------------------------------------------
[2015-05-05 16:13:46] mark at briley dot com
Description:
------------
---
From manual page: http://www.php.net/function.array-unique
---
This problem seems to affect several functions. Array-unique, sort, and others that work with
arrays.
Problem: I had 50MB of INSERT commands which were computer generated. There were several duplicates
as updates had come in to the data set before I ran the program to generate the INSERT commands.
All INSERT commands are made the same way. Thus, the first entry in the INSERT command was the ID
of the record. Duplicates are not allowed so these extra INSERT commands had to be removed. Using
PHP's built-in SORT() command did not sort the records. At first I thought I had done
something wrong and tried it on a smaller data set. Sort worked on the smaller data set but
silently fails on the larger (50MB) data set. Tried array_unique as well as several other array
commands. It seems that when the data set gets over a certain size these functions simple return
the original array. In memory, the program consumed over 2GB of memory. My machine has 8GB of
memory and over 100GB of disk space.
Test script:
---------------
I can not provide a test script since you would need over 50MB of data with records reaching a
length of over 65K characters. By the way - the Windows SORT command can not handle over 65K length
strings. So if PHP calls the Windows SORT routine - that might be the problem.
In my program, I had a small section that simply said:
# $ary = the 50MB list of INSERT commands.
$b = array_unique( $ary );
# $b should now have an array that is unique. But when written out to
# items.sql - there were duplicate INSERT commands. When I compared
# these INSERT commands they were identical. All lines were sent
# through TRIM() to ensure no invisible characters were in a line.
With the SORT routine, I did
# $ary = the 50MB list of INSERT commands.
sort( $ary );
for( $i=0; $i<count($ary)-1; $i++ ){
if( $ary[$i] === $ary[$i+1] ){ unset( $ary[$i] ); }
}
$ary = array_reverse( array_reverse( $ary ) );
$ary still contained duplicates. The weird thing was - the duplicates were hundreds of records
apart. So maybe at record #567 and record #1245. Or even #34629. The sort function should have
put them all next to each other. After all, if the records are:
INSERT INTO <table> (id,atr1,atr2,atr3...) VALUES
(1,"hair","face","feet",...)
and that first value (ie: 1) is on two different records - then they should sort and be together.
This is even if the first value was something like 11256. Because "1," would sort either
before or after "11256," for ALL occurrances of "1,".
Expected result:
----------------
If any of the routines can not handle what is given to it I would have expected an error to be
generated. Anything to let the programmer know there is a problem. This has taken over a week to
track down because there are no errors generated but when I tried to import the file into MySQL I
received a "Duplicate entry..." message which is the only error I ever got letting me know
there was a problem.
Actual result:
--------------
The items.sql file was rife with duplicate entries. When compared by hand the records were exactly
the same. No duplicates should have been possible.
------------------------------------------------------------------------
--
Edit this bug report at https://bugs.php.net/bug.php?id=69575&edit=1