Bug #80158 [Fbk]: Memory leak when xml nodes are stored into local array
Edit report at https://bugs.php.net/bug.php?id=80158&edit=1
ID: 80158
Updated by: nikic@php.net
Reported by: efbiaiinzinz at hotmail dot com
Summary: Memory leak when xml nodes are stored into local
array
Status: Feedback
Type: Bug
Package: SimpleXML related
Operating System: Linux
PHP Version: 7.4.10
Block user comment: N
Private report: N
New Comment:
When I try your code from the last comment, I get:
429864
429864
429864
429928
I.e. the memory usage stays about the same. I checked this on PHP 7.3, 7.4 and 8.0, and the behavior
was the same for all.
Previous Comments:
------------------------------------------------------------------------
[2020-10-03 09:13:29] efbiaiinzinz at hotmail dot com
Ok, found the trigger of the leak after reducing code step by step.
function generate($counter) {
$file = tempnam(sys_get_temp_dir(), 'xml');
$f = fopen($file, 'wb');
fwrite($f, '<?xml version="1.0"
encoding="UTF-8"?><items>' . "\n");
for ($i = 1; $i <= $counter; $i++)
fwrite($f, '<item id="' . $i . '"><id>' . $i .
'</id><name>item ' . $i . '</name></item>' .
"\n");
fwrite($f, '</items>');
fclose($f);
return $file;
}
function testFile($file) {
$xml = simplexml_load_file($file);
foreach ($xml->item as $item)
doLeak($item->name);
}
function doLeak($obj) {
$obj .= ''; //this causes leak
return (string)$obj;
}
$file = generate(50000);
$mem1 = memory_get_usage();
testFile($file);
$mem2 = memory_get_usage();
testFile($file);
$mem3 = memory_get_usage();
testFile($file);
$mem4 = memory_get_usage();
echo "$mem1\n$mem2\n$mem3\n$mem4\n";
unlink($file);
Every execution sees memory usage increasing permanently.
------------------------------------------------------------------------
[2020-10-03 07:19:17] efbiaiinzinz at hotmail dot com
I'm having bit of a trouble trying to debug it more due to the
"harmless" leak because it makes it hard to pinpoint exact place and
behaviour where leak is occurring.
I confirmed overnight that this leak issue also exists in 7.3.22.
I modified the sample script and changed:
$mem1 = memory_get_usage();
testFile($file);
$mem2 = memory_get_usage();
echo "$mem1\n$mem2\n";
to:
$mem1 = memory_get_usage();
testFile($file);
$mem2 = memory_get_usage();
$str500KB = str_repeat('1234567890', 50 * 1024);
$mem3 = memory_get_usage();
$str10MB = str_repeat('1234567890', 1024 * 1024);
$mem4 = memory_get_usage();
testFile($file);
$mem5 = memory_get_usage();
echo "$mem1\n$mem2\n$mem3\n$mem4\n$mem5\n";
I see output (running through apache)
391936
908032
1424128
11914008
11914008
Creating new strings does not reuse the memory but rather increases the
memory usage on top of existing number, but rerunning the same simplexml
elemnt parser + loop does not increase the total memory indeed.
Do you have any pointers on how to debug it more or if there is some
known edge case that can cause the object store harmless leak turn into
an actual leak?
Right now I have one codepath with xml input of 50MB and according to
memory_get_usage() running it twice in row results ~72MB getting lost
for each run and another codepath that uses 80MB json file as input and
first run shows memory difference 0.11MB (loads multiple classes) and
second run shows memory difference 0.00MB as expected.
I will try to narrow it down some more this weekend.
------------------------------------------------------------------------
[2020-10-02 12:05:47] efbiaiinzinz at hotmail dot com
I'll try to make a more complex sample, but something strange is happening still
.
The full system where it started occurring does lots of reading from different datapoints (some xml,
some json, some csv).
There is a common Base class extended by different per-endpoint implementations, that map the
specific structure into common internal format and then pass it on to additional handlers that store
changes into db.
I have added logging and the handlers that use bigger CSV files (50MB plus) and read 100k rows and
do lots of interactions show memory differece 0.01-0.05MB (class definition loading takes memory).
When measuring memory after XML parsers, they all report memory "leag" counted in
megabytes, worst case 80MB+
And out of memory has happened due to it already, so it seems to be eprmanent leak affecting new
allocations.
I'll try to run the detailed measurements when running cron with 7.3 version and then can
report the difference in more detail.
------------------------------------------------------------------------
[2020-10-01 14:50:01] nikic@php.net
> When looping over xml nodes and storing the node objects temporarily into a local array, some
> memory is never released and doing it many times in a row with different files will result in out of
> memory error.
From a cursory look, this looks like an object store "leak", which means that memory usage
is proportional to the maximum number of objects live at the same time. This is generally harmless
as it does not result in a persistent leak and the memory is reused when creating new objects.
However, it also means that "doing it many times in a row with different files will result in
out of memory error" does not hold, assuming you don't keep objects alive between files.
------------------------------------------------------------------------
[2020-09-27 21:34:47] efbiaiinzinz at hotmail dot com
Description:
------------
When looping over xml nodes and storing the node objects temporarily into a local array, some memory
is never released and doing it many times in a row with different files will result in out of memory
error.
Provided sample shows that ~516096 bytes is lost.
When I change the test script line
$items[(int)(string)$item->id] = $item;
to
$items[(int)(string)$item->id] = (string)$item->name;
or even to
$items[(int)(string)$item->id] = $item->asXML();
Then memory amount before and after testFile() call is exactly the same.
Test script:
---------------
function generate($counter) {
$file = tempnam(sys_get_temp_dir(), 'xml');
$f = fopen($file, 'wb');
fwrite($f, '<?xml version="1.0"
encoding="UTF-8"?><items>' . "\n");
for ($i = 1; $i <= $counter; $i++)
fwrite($f, '<item id="' . $i . '"><id>' . $i .
'</id><name>item ' . $i . '</name></item>' .
"\n");
fwrite($f, '</items>');
fclose($f);
return $file;
}
function testFile($file) {
$xml = simplexml_load_file($file);
$items = [];
foreach ($xml->item as $item)
$items[(int)(string)$item->id] = $item;
}
$file = generate(50000);
$mem1 = memory_get_usage();
testFile($file);
$mem2 = memory_get_usage();
echo "$mem1\n$mem2\n";
unlink($file);
Expected result:
----------------
The test should print two exactly same numbers, separated by linebreak.
Actual result:
--------------
Second number is ~516 000 larger which means at least 0.5MB of memory leaked somehow.
------------------------------------------------------------------------
--
Edit this bug report at https://bugs.php.net/bug.php?id=80158&edit=1
Thread (6 messages)