Bug #78987 [Com]: Memory problems running finfo::buffer with PHP_CLI

Edit report at https://bugs.php.net/bug.php?id=78987&edit=1 ID: 78987 Comment by: magnar at myrtveit dot com Reported by: jhhillie at amazon dot com Summary: Memory problems running finfo::buffer with PHP_CLI Status: Verified Type: Bug Package: Filesystem function related Operating System: Ubuntu PHP Version: 7.4Git-2019-12-18 (Git) Block user comment: N Private report: N New Comment: I've created two scripts that illustrate the problem with finfo_buffer: Memory usage of finfo_file: https://3v4l.org/M3YLG Memory usage of finfo_buffer: https://3v4l.org/LNNoV On a 16M file, finfo_file uses 20M memory. finfo_buffer, however, uses 244M! Previous Comments: ------------------------------------------------------------------------ [2020-02-13 08:15:27] nikic@php.net Related To: Bug #79263 ------------------------------------------------------------------------ [2019-12-18 10:58:00] nikic@php.net I can reproduce the large allocation. It is caused by https://github.com/php/php-src/blob/aadd5e69e01728adb78d03026beb9a9a0c7e75ef/ext/fileinfo/libmagic/encoding.c#L92. Basically, libmagic tries to detect the encoding of the file and does this by ... allocating a buffer of unicode characters for the whole file. For some reason this also uses "unsigned long" instead of "uint32_t" for each character, which means the allocation is not just 4 times as large, but 8 times as large as the original file! Current upstream version still looks about the same: https://github.com/file/file/blob/master/src/encoding.c Ideally this issue would be reported upstream and fixed there first. ------------------------------------------------------------------------ [2019-12-18 04:51:12] requinix@php.net > Failed identify data 12:cannot allocate 2516582408 bytes (Cannot allocate > memory)application/octet-stream According to that, libmagic is the one failing. Working for me with PHP 7.0-7.4 on Ubuntu. Does it actually include "application/octet-stream" in there? That is the correct MIME type... I assume the system is, in fact, running out of memory? What version of libmagic? ------------------------------------------------------------------------ [2019-12-18 04:12:28] jhhillie at amazon dot com Description: ------------ When calling finfo::buffer() with a crafted string, PHP tries to allocate an insane amount of memory to do so. I found a similar bug here - https://bugs.php.net/bug.php?id=68819 I suspect the bug may have been re-introduced at some stage during newer versions. 1) I tested it on the version my customer had been using - v7.0.33 (doesn't work) 2) I then tested on the latest php version - v7.4.0 (doesn't work) 3) Testing on an older version - v5.4.16 it works perfectly. Test script: --------------- Generated a random file of 300MB using: $ sudo fallocate -l 300M test-file The script used: <?php $content = file_get_contents('test-file', true); $fileInfo = new \finfo(FILEINFO_MIME_TYPE); var_dump($fileInfo->buffer($content)); ?> Expected result: ---------------- To provide the file metadata (I know there are other ways to do this): string(19) "application/x-empty" Actual result: -------------- PHP Warning: finfo::buffer(): Failed identify data 12:cannot allocate 2516582408 bytes (Cannot allocate memory)application/octet-stream in php shell code on line 1 bool(false) ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=78987&edit=1