Bug #78987 [Com]: Memory problems running finfo::buffer with PHP_CLI

Edit report at https://bugs.php.net/bug.php?id=78987&edit=1 ID: 78987 Comment by: halaeiv at gmail dot com Reported by: jhhillie at amazon dot com Summary: Memory problems running finfo::buffer with PHP_CLI Status: Verified Type: Bug Package: Filesystem function related Operating System: Ubuntu PHP Version: 7.4Git-2019-12-18 (Git) Block user comment: N Private report: N New Comment: I have reported to bug to file package https://bugs.astron.com/view.php?id=234 and they fixed it really quickly: "file_buffer(3) passed the full size of the buffer to the encoding determination function. If the file was too large, we ended up allocating (2 * size + 4 * size) buffers to scan for encoding. Now we limit size to 64K." Can we have this fix in PHP soon? Thanks Previous Comments: ------------------------------------------------------------------------ [2021-02-04 11:00:18] halaeiv at gmail dot com I have a quick question. If it is libmagic bug how come the bug exists in php-7.4 but not php-7.2? Both are installed via ondrej/php in Ubuntu and both packages depend on the same libmagic1. If 2 php versions depends on the same shared library and one has a bug, doesn't this mean the bug is actually not in the shared library? ------------------------------------------------------------------------ [2020-06-12 07:12:11] magnar at myrtveit dot com I've created two scripts that illustrate the problem with finfo_buffer: Memory usage of finfo_file: https://3v4l.org/M3YLG Memory usage of finfo_buffer: https://3v4l.org/LNNoV On a 16M file, finfo_file uses 20M memory. finfo_buffer, however, uses 244M! ------------------------------------------------------------------------ [2020-02-13 08:15:27] nikic@php.net Related To: Bug #79263 ------------------------------------------------------------------------ [2019-12-18 10:58:00] nikic@php.net I can reproduce the large allocation. It is caused by https://github.com/php/php-src/blob/aadd5e69e01728adb78d03026beb9a9a0c7e75ef/ext/fileinfo/libmagic/encoding.c#L92. Basically, libmagic tries to detect the encoding of the file and does this by ... allocating a buffer of unicode characters for the whole file. For some reason this also uses "unsigned long" instead of "uint32_t" for each character, which means the allocation is not just 4 times as large, but 8 times as large as the original file! Current upstream version still looks about the same: https://github.com/file/file/blob/master/src/encoding.c Ideally this issue would be reported upstream and fixed there first. ------------------------------------------------------------------------ [2019-12-18 04:51:12] requinix@php.net > Failed identify data 12:cannot allocate 2516582408 bytes (Cannot allocate > memory)application/octet-stream According to that, libmagic is the one failing. Working for me with PHP 7.0-7.4 on Ubuntu. Does it actually include "application/octet-stream" in there? That is the correct MIME type... I assume the system is, in fact, running out of memory? What version of libmagic? ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=78987 -- Edit this bug report at https://bugs.php.net/bug.php?id=78987&edit=1