Bug #81585 [Asn->Ana]: Reused cached_chunks are not counted to heap size

Edit report at https://bugs.php.net/bug.php?id=81585&edit=1 ID: 81585 Updated by: cmb@php.net Reported by: xtpd17 at gmail dot com -Summary: Memory leaks when creating short string arrrays +Summary: Reused cached_chunks are not counted to heap size -Status: Assigned +Status: Analyzed Type: Bug -Package: Apache related +Package: Scripting Engine problem -Operating System: Windows +Operating System: * PHP Version: 7.4 Assigned To: cmb Block user comment: N Private report: N New Comment: Setting ThreadsPerChild 1 actually yields only 1 worker thread; the other threads are ancilliary. Anyhow, the problem is that we don't count reused cached_chunks (i.e. those which are retained between requests) to the size of the heap, so the memory stats are wrong, and the memory_limit is not properly heeded. The latter causes the reported memory leak, because the script is trying to exploit that. This is not particularly related to Windows and Apache, but happens for all web environments, i.e. whenever a process or thread serves multiple consecutive requests (and chunks are cached). Previous Comments: ------------------------------------------------------------------------ [2021-11-29 17:53:28] cmb@php.net First, the fact that not all request memory is freed on shutdown, is a deliberate design decisions[1]. The reasoning is that later reallocation of the memory from the OS is expensive. Second, the total amount of memory allocated (httpd_mb) can easily exceed PHP's memory_limit for a multi-threaded environment such as Apache mpm_winnt. If I set ThreadsPerChild 1 (note there is only one child for mpm_winnt), I still get four threads (might be a hard coded limit). Then the maximum amount of memory allocated is about four times the memory_limit, so this doesn't look like a bug. However, the reported values for memory_get_usage() are not what I would have expected; it seems that memory_get_usage(true) reports the per thread values, but memory_get_usage(false) return the process values. I'll have a closer look. [1] <https://github.com/php/php-src/blob/php-7.4.26/Zend/zend_alloc.c#L2264> ------------------------------------------------------------------------ [2021-11-01 19:59:52] xtpd17 at gmail dot com Description: ------------ (System: Win7x64, last Apache 2.4 x32, php x32) If your script creates array of small strings (not more than 2mb approx.) and gets all amount of available memory (memory_limit could be set or left "-1") and then stops on "Fatal error: Allowed memory exhausted", it doesn't free all memory, so size of httpd.exe remains huge. If you run this script several times, size of httpd.exe grows and grows, and at some point your _another_ "innocent" scripts fails when trying to alloc relatively small chunk of memory with "Out of memory" (though their limits were not exceeded). Steps to reproduce: reload script in browser several times until it fails. You'll see growing initial size of httpd, then fail. Notable things: - memory leaks only when we create small string (<2mb) - big (or not set) memory_limit spends memory faster - it looks like E_USER_ERROR also keeps memory busy (see commented strings in script) - at second run memory_get_usage(true) is close to zero, memory_get_usage() grows. - in fact we have "out of memory" error already in the second run, but I thought it would be more clear to have separate block to demonstrate leak. - feel free to experiment with settings values in script. TEST RESULTS (memory_limit = 1500M): FIRST RUN ---------------------- apache size: 16 mb, memory usage/real usage: 0/2 mb (0 items in array, 0 mb stored) apache size: 115 mb, memory usage/real usage: 98/100 mb (49 items in array, 98 mb stored) apache size: 213 mb, memory usage/real usage: 196/198 mb (98 items in array, 196 mb stored) ... Fatal error: Allowed memory size of 1572864000 bytes exhausted (tried to allocate 2093056 bytes) in test_leak.php on line 58 before crash: apache size: 1517 mb, memory usage/real usage: 1495/1500 mb (749 items in array, 1494 mb stored) --- SECOND RUN ------------------------------ apache size: 766 mb, memory usage/real usage: 0/2 mb (0 items in array, 0 mb stored) apache size: 766 mb, memory usage/real usage: 98/2 mb (49 items in array, 98 mb stored) apache size: 766 mb, memory usage/real usage: 196/2 mb (98 items in array, 196 mb stored) /// Fatal error: Out of memory (allocated 1113587712) (tried to allocate 2093056 bytes) in test_leak.php on line 58 (WE HAVE OUT OF MEMORY ERROR ALREADY AT THIS POINT) before crash: apache size: 1828 mb, memory usage/real usage: 1805/1062 mb (904 items in array, 1804 mb stored) --- THEN YOU GET 1297 mb of apache, then 1562... --- AND THEN... FAIL RUN ------------------------------------- apache size: 1694 mb, memory usage/real usage: 0/2 mb (0 items in array, 0 mb stored) apache overloaded (1694 mb), trying to alloc 200 mb Fatal error: Out of memory (allocated 2097152) (tried to allocate 209715224 bytes) in test_leak.php on line 51 before crash: apache size: 1694 mb, memory usage/real usage: 0/2 mb (0 items in array, 0 mb stored) Test script: --------------- https://pastebin.com/qh8hneTg ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=81585&edit=1