Bug #69575 [NEW]: Mega data - mega problem

From: mark at briley dot com Operating system: Windows 7 Pro PHP version: 5.5.24 Package: *General Issues Bug Type: Bug Bug description:Mega data - mega problem Description: ------------ --- From manual page: http://www.php.net/function.array-unique --- This problem seems to affect several functions. Array-unique, sort, and others that work with arrays. Problem: I had 50MB of INSERT commands which were computer generated. There were several duplicates as updates had come in to the data set before I ran the program to generate the INSERT commands. All INSERT commands are made the same way. Thus, the first entry in the INSERT command was the ID of the record. Duplicates are not allowed so these extra INSERT commands had to be removed. Using PHP's built-in SORT() command did not sort the records. At first I thought I had done something wrong and tried it on a smaller data set. Sort worked on the smaller data set but silently fails on the larger (50MB) data set. Tried array_unique as well as several other array commands. It seems that when the data set gets over a certain size these functions simple return the original array. In memory, the program consumed over 2GB of memory. My machine has 8GB of memory and over 100GB of disk space. Test script: --------------- I can not provide a test script since you would need over 50MB of data with records reaching a length of over 65K characters. By the way - the Windows SORT command can not handle over 65K length strings. So if PHP calls the Windows SORT routine - that might be the problem. In my program, I had a small section that simply said: # $ary = the 50MB list of INSERT commands. $b = array_unique( $ary ); # $b should now have an array that is unique. But when written out to # items.sql - there were duplicate INSERT commands. When I compared # these INSERT commands they were identical. All lines were sent # through TRIM() to ensure no invisible characters were in a line. With the SORT routine, I did # $ary = the 50MB list of INSERT commands. sort( $ary ); for( $i=0; $i<count($ary)-1; $i++ ){ if( $ary[$i] === $ary[$i+1] ){ unset( $ary[$i] ); } } $ary = array_reverse( array_reverse( $ary ) ); $ary still contained duplicates. The weird thing was - the duplicates were hundreds of records apart. So maybe at record #567 and record #1245. Or even #34629. The sort function should have put them all next to each other. After all, if the records are: INSERT INTO <table> (id,atr1,atr2,atr3...) VALUES (1,"hair","face","feet",...) and that first value (ie: 1) is on two different records - then they should sort and be together. This is even if the first value was something like 11256. Because "1," would sort either before or after "11256," for ALL occurrances of "1,". Expected result: ---------------- If any of the routines can not handle what is given to it I would have expected an error to be generated. Anything to let the programmer know there is a problem. This has taken over a week to track down because there are no errors generated but when I tried to import the file into MySQL I received a "Duplicate entry..." message which is the only error I ever got letting me know there was a problem. Actual result: -------------- The items.sql file was rife with duplicate entries. When compared by hand the records were exactly the same. No duplicates should have been possible. -- Edit bug report at https://bugs.php.net/bug.php?id=69575&edit=1 -- Try a snapshot (PHP 5.4): https://bugs.php.net/fix.php?id=69575&r=trysnapshot54 Try a snapshot (PHP 5.5): https://bugs.php.net/fix.php?id=69575&r=trysnapshot55 Try a snapshot (trunk): https://bugs.php.net/fix.php?id=69575&r=trysnapshottrunk Fixed in SVN: https://bugs.php.net/fix.php?id=69575&r=fixed Fixed in release: https://bugs.php.net/fix.php?id=69575&r=alreadyfixed Need backtrace: https://bugs.php.net/fix.php?id=69575&r=needtrace Need Reproduce Script: https://bugs.php.net/fix.php?id=69575&r=needscript Try newer version: https://bugs.php.net/fix.php?id=69575&r=oldversion Not developer issue: https://bugs.php.net/fix.php?id=69575&r=support Expected behavior: https://bugs.php.net/fix.php?id=69575&r=notwrong Not enough info: https://bugs.php.net/fix.php?id=69575&r=notenoughinfo Submitted twice: https://bugs.php.net/fix.php?id=69575&r=submittedtwice register_globals: https://bugs.php.net/fix.php?id=69575&r=globals PHP 4 support discontinued: https://bugs.php.net/fix.php?id=69575&r=php4 Daylight Savings: https://bugs.php.net/fix.php?id=69575&r=dst IIS Stability: https://bugs.php.net/fix.php?id=69575&r=isapi Install GNU Sed: https://bugs.php.net/fix.php?id=69575&r=gnused Floating point limitations: https://bugs.php.net/fix.php?id=69575&r=float No Zend Extensions: https://bugs.php.net/fix.php?id=69575&r=nozend MySQL Configuration Error: https://bugs.php.net/fix.php?id=69575&r=mysqlcfg