[php-src] Issue #13599: Out of memory with 1MB strings even though memory_get_usage reports 50% utilization

From: Date: Tue, 05 Mar 2024 14:10:29 +0000
Subject: [php-src] Issue #13599: Out of memory with 1MB strings even though memory_get_usage reports 50% utilization
Groups: php.bugs 
Request: Send a blank email to php-bugs+get-246512@lists.php.net to get a copy of this message
Issue: https://github.com/php/php-src/issues/13599 Author: martinhoch42 ### Description I've observed that a script running for an extended period crashes due to a "Fatal error: Allowed memory size" despite the internal memory usage, as reported by memory_get_usage(), being significantly lower (as low as 51% of the memory_limit). Upon further investigation, it appears PHP uses a large amount of additional system memory when managing strings between 1 and 2 MB in size, and possibly for smaller sizes as well. https://gist.github.com/martinhoch42/44185de7f8c18e2742058e8fa7072f71 ``` Start test with Chunk 1 and Count 50. ... Loop 45: | Mem: 46 MB | Real: 90 MB | Cache: 45 MB| Leak%: 1.97 ``` ``` Start test with Chunk 1.5 and Count 50. ... Loop 45: | Mem: 68 MB | Real: 92 MB | Cache: 67.5 MB| Leak%: 1.35 ``` ``` Start test with Chunk 2 and Count 50. ... Loop 45: | Mem: 91 MB | Real: 94 MB | Cache: 90 MB| Leak%: 1.04 ``` Notice the discrepancy between "Mem" and "Real". I understand handling memory is not free and that there are many issues in play here, but I feel this behavior is odd and should be considered a bug since it affects real-world applications. This script showcases the issue within a fictional auto-purging cache: https://gist.github.com/martinhoch42/db9dec5eaf1c6a4e29a79ef0b104df34 It has a cache storing n-sized strings, tracks memory usage, and purges the cache if a specific percentage of memory is utilized, based on memory_get_usage. The script encounters failure under the following conditions: * The size is between 1MB and 2MB. * The limit exceeds 51% for 1MB strings. (Increasing the size allows for a higher limit.) * Sizes above 2MB consistently work. * Sizes below 1MB also cause a problem if the limit is high enough. * Setting the limit below 50% also works. In the most severe scenario, the script fails while using only ~50% of the internal memory: ``` Mem: 257 MB | Real: 512 MB | Cache: 256 MB fatal error: allowed memory size of 536870912 bytes exhausted at ./php-src/Zend/zend_string.h:185 (tried to allocate 1048640 bytes) in realmemoryleak.php on line 67 ``` ### Expectations: The script cachetest.php should never run into "fatal error: allowed memory size ..." when using any reasonable value for size and limit. A 100% difference (for 1MB strings) between memory_get_usage() and memory_get_usage(true) is at the very least odd. I understand PHP can request more system memory than the application currently uses, but the extra memory in this case appears to be "lost". This does not appear to be a leak but rather a fixed overhead of real vs internal memory. ### Additional Information: The behavior exists in at least 8.1, 8.2 and 8.3. Attempting to invoke the garbage collector (gc_collect_cycles) manually within the scripts resulted in no changes. Please let me know if anything is unclear or if I can provide more information. ### PHP Version 8.3 ### Operating System _No response_

« previous php.bugs (#246512) next »