Bug #80384 [PATCH]: filter buffers entire read until file closed
Edit report at https://bugs.php.net/bug.php?id=80384&edit=1
ID: 80384
Patch added by: adamjseitz@gmail.com
Reported by: adamjseitz at gmail dot com
Summary: filter buffers entire read until file closed
Status: Verified
Type: Bug
Package: Filter related
Operating System: Debian Buster on WSL2
PHP Version: 7.4.12
Block user comment: N
Private report: N
New Comment:
The following pull request has been associated:
Patch Name: Fix #80384: limit read buffer size
On GitHub: https://github.com/php/php-src/pull/6444
Patch: https://github.com/php/php-src/pull/6444.patch
Previous Comments:
------------------------------------------------------------------------
[2020-11-20 15:05:26] cmb@php.net
This is not particularly related to zlib.inflate, but rather a
general issue for streams which have any filter attached.
php_stream_fill_read_buffer()[1] only reads a single chunk if no
filter is attached, but the full size if a filter is attached.
Frankly, it's not clear to me why we're looping[2] inside that
function.
[1] <https://github.com/php/php-src/blob/php-7.4.12/main/streams/streams.c#L540>
[2] <https://github.com/php/php-src/blob/php-7.4.12/main/streams/streams.c#L552>
------------------------------------------------------------------------
[2020-11-20 04:14:52] adamjseitz at gmail dot com
Description:
------------
"zlib.inflate" filters appear to buffer all previously-read data, as exhibited by the
attached code, until the file is closed.
It is unclear from my testing if the data being kept in memory is the original or inflated copy.
Test script:
---------------
<?php
// Generate "data.gz" file with:
// dd if=/dev/urandom of=data count=$((8*1024*1024)) iflag=count_bytes; gzip data
function PrintMem($message) { echo $message, ": ", (memory_get_usage() / 1024 / 1024) .
" MB\n"; }
$fp = fopen("data.gz", 'rb');
$filter = stream_filter_append($fp, "zlib.inflate", STREAM_FILTER_READ, array(
'window' => 31 ));
PrintMem("before read");
stream_get_contents($fp);
PrintMem("after read");
stream_filter_remove($filter);
fclose($fp);
PrintMem("after close");
?>
Expected result:
----------------
I expect that very little memory is used.
It should be possible to read smaller blocks at a time from a very large gzipped files without
buffering the entire content of what has been read so far, but that does not seem to be the case.
Actual result:
--------------
OUTPUT from the above script:
before read: 0.44554138183594 MB
after read: 8.4377517700195 MB
after close: 0.37459564208984 MB
------------------------------------------------------------------------
--
Edit this bug report at https://bugs.php.net/bug.php?id=80384&edit=1
Thread (5 messages)