Bug #79143 [Opn->Dup]: Memory leak with streams and HTTP protocol 1.1

Edit report at https://bugs.php.net/bug.php?id=79143&edit=1 ID: 79143 Updated by: nikic@php.net Reported by: curtisfarnham at gmail dot com Summary: Memory leak with streams and HTTP protocol 1.1 -Status: Open +Status: Duplicate Type: Bug Package: Streams related Operating System: Ubuntu, Alpine, MacOS PHP Version: 7.4.1 Block user comment: N Private report: N New Comment: Looks like the root cause here was the same as for bug #78902, which is now fixed. Previous Comments: ------------------------------------------------------------------------ [2020-01-20 20:58:39] nikic@php.net Sorry, I meant "auto_decode" => false of course. ------------------------------------------------------------------------ [2020-01-20 20:58:06] nikic@php.net Relevant massif: ->08.21% (338,643B) 0x903803: __zend_realloc (zend_alloc.c:2994) | ->08.15% (335,955B) 0x90278E: _realloc_custom (zend_alloc.c:2434) | | ->08.12% (334,923B) 0x9028DE: _erealloc (zend_alloc.c:2556) | | | ->08.08% (333,347B) 0x8BEE2C: _php_stream_fill_read_buffer (streams.c:597) | | | | ->08.08% (333,347B) 0x8BF3CE: _php_stream_read (streams.c:706) | | | | ->08.08% (333,347B) 0x8BF548: php_stream_read_to_str (streams.c:752) | | | | ->08.08% (333,347B) 0x74E5D0: zif_fread (file.c:1810) | | | | ->08.08% (333,347B) 0x9B0767: ZEND_DO_ICALL_SPEC_RETVAL_USED_HANDLER (zend_vm_execute.h:1314) | | | | ->08.08% (333,347B) 0xA164B4: execute_ex (zend_vm_execute.h:53797) | | | | ->08.08% (333,347B) 0xA1A5F3: zend_execute (zend_vm_execute.h:57913) | | | | ->08.08% (333,347B) 0x93DA8B: zend_execute_scripts (zend.c:1665) | | | | ->08.08% (333,347B) 0x89F734: php_execute_script (main.c:2617) | | | | ->08.08% (333,347B) 0xA1D1DD: do_cli (php_cli.c:961) | | | | ->08.08% (333,347B) 0xA1E358: main (php_cli.c:1352) Possible relevant difference is the dechunking stream filter. Indeed, setting "http" => "auto_decode" => true avoids the issue. ------------------------------------------------------------------------ [2020-01-20 20:41:30] curtisfarnham at gmail dot com Description: ------------ When using the "http" stream context and protocol 1.1, there is a substantial memory leak which begins after 2.00 MB of a stream has been downloaded. It increases linearly with time and is roughly in proportion to the amount of data downloaded. This bug seems to have been introduced in PHP 7.2 but is still there in 7.4.1. I have experienced it on Alpine Linux, Ubuntu Linux, and MacOS/Homebrew. Using Protocol 1.0 successfully avoids the leak. Strangely, it also doesn't occur with MacOS Catalina's PHP 7.3.9 binary. This discussion began at https://github.com/guzzle/guzzle/issues/2555 . Since this test script discards the contents of the stream, the memory usage should hold constant over time. It's initially constant, but when it gets to a cumulative 2097152 bytes downloaded (2.00 MB), the bug begins to appear and memory starts increasing steadily. Also, there are two stepwise increases in memory usage, each of exactly 2.00 MB, prior to when the script reaches the aforementioned threshold. The increases occur at different times depending on the specific environment. (Credit goes to Tobias Nyholm and Konstantin Kopachev for helping troubleshoot this. Thanks!) Test script: --------------- <?php $context = stream_context_create(['http' => [ 'protocol_version' => '1.1', ]]); $stream = fopen('https://kpbs.streamguys1.com/?ck=1579203064304', 'r', null, $context); $dl = 0; while (!feof($stream)) { $chunk = fread($stream, 8192); $dl += strlen($chunk); echo "downloaded $dl, memory " . memory_get_usage(true) . "\n"; } Expected result: ---------------- downloaded 8192, memory 2097152 downloaded 16384, memory 2097152 downloaded 24576, memory 2097152 ... downloaded 1114112, memory 2097152 downloaded 1122304, memory 2097152 downloaded 1130496, memory 2097152 downloaded 1138688, memory 2097152 ... downloaded 2080768, memory 2097152 downloaded 2088960, memory 2097152 downloaded 2097152, memory 2097152 downloaded 2105344, memory 2097152 downloaded 2113536, memory 2097152 downloaded 2121728, memory 2097152 Actual result: -------------- downloaded 8192, memory 2097152 downloaded 16384, memory 2097152 downloaded 24576, memory 2097152 ... downloaded 1114112, memory 2097152 downloaded 1122304, memory 2097152 downloaded 1130496, memory 4194304 downloaded 1138688, memory 4194304 ... downloaded 2080768, memory 4194304 downloaded 2088960, memory 4194304 downloaded 2097152, memory 6295552 downloaded 2105344, memory 6303744 downloaded 2113536, memory 6311936 downloaded 2121728, memory 6320128 ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=79143&edit=1