#45291 [Opn->Bgs]: Memory failure in very large XML file

ID: 45291 Updated by: rrichards@php.net Reported By: gilbert dot musnik at fr dot adp dot com -Status: Open +Status: Bogus Bug Type: SimpleXML related Operating System: Windows PHP Version: 5.2.6 New Comment: Thank you for taking the time to write to us, but this is not a bug. Please double-check the documentation available at http://www.php.net/manual/ and the instructions on how to report a bug at http://bugs.php.net/how-to-report.php Memory for xml is allocated by third party library and not part of PHP memory usage controls. If you don't know/can't control size of the xml files, you should use streaming access (xmlreader or ext/xml) and not dom based (simplexml or dom). Previous Comments: ------------------------------------------------------------------------ [2008-06-17 08:04:41] gilbert dot musnik at fr dot adp dot com Description: ------------ Hello, It's been days I am trying to sort out a problem which pops up when I process a huge XML file. In "reproduce code" windows is a trivial script which allocates memory by making an array very big. Assuming a memory_limit of 1024M in <php.ini>, if I don't load the huge XML file, the script crashes as expected with a "memory exhausted" fatal error diagnostic (see expected results window). However, if I do load my huge XML file with simplexml_load_file(around 300 MB!), the script crashes with a "Out of memory" fatal error diagnostic. By the way, it doesn't matter if memory_limit is set to 512M or 2048M, the script stumbles on the same "allocated" figure of 332 MB (see actual results windows). The strangest thing is that before I start swelling the array, the memory usage is almost the same whether ot not simplexml_load_file("VERY_BIG_FILE.XML") is invoked or not. It looks as if memory allocated by simplexml_load_file() routine is NOT taken into account in overall PHP memory usage and memory limit management. Thanks in advance for your feedback on that, Gilbert Musnik Reproduce code: --------------- <?php $flag_xml = true; $now = Date("Y/m/d - H:i:s"); if ($flag_xml) { print ("[ $now ] Before simplexml_load_file\n"); $f_inp = "VERY_BIG_FILE.xml"; // mine is 300 MB $ll0 = time(); $xml = simplexml_load_file ($f_inp); $ll1 = time() - $ll0; $now = Date("Y/m/D - H:i:s"); print ("[ $now ] After simplexml_load_file: $ll1 secondes\n"); } else { print ("[ $now ] NO simplexml_load_file\n"); } $bufrs = array(); $nmax = 150000000; for ($ii = 0; $ii < $nmax; $ii ++) { $bufr = str_repeat ("ABCDEFGHIJ", 1); array_push ($bufrs, $bufr); $jj = $ii % 1000000; if ($jj == 0) { $xx = number_format ($ii); $qq = number_format (memory_get_usage()); $memory_limit = ini_get('memory_limit'); print ("$xx - memory usage=$qq - memory_limit=$memory_limit)\n"); } }; ?> Expected result: ---------------- N:\>\php-5.2.6-Win32\php.exe -c \MD_config\php.ini \MD_config\test_mem.php [ 2008/06/16 - 18:06:10 ] No simplexml_load_file 0 - memory usage=64,552 - memory_limit=1024M) 1,000,000 - memory usage=100,259,664 - memory_limit=1024M) 2,000,000 - memory usage=200,454,016 - memory_limit=1024M) 3,000,000 - memory usage=304,842,624 - memory_limit=1024M) 4,000,000 - memory usage=400,842,624 - memory_limit=1024M) 5,000,000 - memory usage=513,619,840 - memory_limit=1024M) 6,000,000 - memory usage=609,619,840 - memory_limit=1024M) 7,000,000 - memory usage=705,619,840 - memory_limit=1024M) 8,000,000 - memory usage=801,619,840 - memory_limit=1024M) 9,000,000 - memory usage=931,174,272 - memory_limit=1024M) 10,000,000 - memory usage=1,027,174,272 - memory_limit=1024M) Fatal error: Allowed memory size of 1073741824 bytes exhausted (tried to allocate 35 bytes) in N:\md_config\test_mem.php on line 23 Actual result: -------------- N:\>\php-5.2.6-Win32\php.exe -c \MD_config\php.ini \MD_config\test_mem.php [ 2008/06/16 - 18:05:06 ] Before simplexml_load_file [ 2008/06/Mon - 18:05:33 ] After simplexml_load_file: 27 secondes 0 - memory usage=65,472 - memory_limit=1024M) 1,000,000 - memory usage=100,260,376 - memory_limit=1024M) 2,000,000 - memory usage=200,454,728 - memory_limit=1024M) 3,000,000 - memory usage=304,843,336 - memory_limit=1024M) Fatal error: Out of memory (allocated 332660736) (tried to allocate 35 bytes) in N:\md_config\test_mem.php on line 23 ------------------------------------------------------------------------ -- Edit this bug report at http://bugs.php.net/?id=45291&edit=1