Edit report at https://bugs.php.net/bug.php?id=69575&edit=1
ID: 69575
User updated by: mark at briley dot com
Reported by: mark at briley dot com
Summary: Mega data - mega problem
-Status: Feedback
+Status: Open
Type: Bug
Package: *General Issues
Operating System: Windows 7 Pro
PHP Version: 5.5.24
Block user comment: N
Private report: N
New Comment:
Give me a day. I'll post something as soon as I have it.
Previous Comments:
------------------------------------------------------------------------
[2015-05-06 21:33:22] cmb@php.net
> The program never got over 3GB but my PHP script never
> complained about the 6000M memory_limit.
That might be regarded as bug, but that's another issue than
array_unique() failing for large arrays, and as such should be
reported as distinct issue.
Anyhow, I wonder how you have been able to run the supplied test
script. All available official Windows builds (x86 as well as x64)
of PHP 5.5.24 don't seem to be able to allocate more than 2G of
memory. Even with a somewhat recent snapshot of an x64 build of
PHP 7.0.0 I have not been able to run the test script (a memory
limit of 6,000,000,000 bytes doesn't suffice, and it seems that
it's not possible to raise the memory limit beyond 8,000,000,000 â
yet another issue).
Can you please give a test script that makes it possible to
reproduce the issue?
------------------------------------------------------------------------
[2015-05-06 18:30:51] mark at briley dot com
Hah! :-) Yeah. The program never got over 3GB but my PHP script never complained about the 6000M
memory_limit. Strange that. :-?
------------------------------------------------------------------------
[2015-05-06 17:24:06] cmb@php.net
Oops! Disregard my last comment; that was nonsense (I had
decreased the number of strings).
------------------------------------------------------------------------
[2015-05-06 17:15:20] cmb@php.net
Gee! Setting the memory_limit to '6000000000' instead of '6000M'
did work (so disregard my former comment about a general 2G memory
limit), and the test script gave the expected results.
------------------------------------------------------------------------
[2015-05-06 17:06:23] cmb@php.net
Hmm, I can't run the given test script, because there is a 2 GB
memory limit in 5.5.24 (x64 as well as experimental x64 build) on
Windows (besides a typo in the script: $strings[$v] should be
$strings[$k]). If I'm not mistaken this memory limit applies to
all current PHP versions.
This might have actually been the cause of your script misbehaving.
| we are taking thousands of XML files, putting them together,
| converting them to SQL, ensuring they are going into the
| database in the right order (sorting), and then splitting the
| SQL file up into separate files.
It might be a workaround to split the big SQL first, then sorting
each file separately, and then doing a merge sort[1] on the files.
[1] <http://en.wikipedia.org/wiki/Merge_sort>
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=69575
--
Edit this bug report at https://bugs.php.net/bug.php?id=69575&edit=1