Bug #77308 [Opn]: Unbuffered queries memory leak

Edit report at https://bugs.php.net/bug.php?id=77308&edit=1 ID: 77308 Updated by: nikic@php.net Reported by: sjon at hortensius dot net Summary: Unbuffered queries memory leak Status: Open Type: Bug Package: MySQL related Operating System: archlinux PHP Version: 7.3.0 -Assigned To: +Assigned To: dmitry Block user comment: N Private report: N New Comment: @dmity: Probably related to the mysqlnd arena improvements. Massif trace: 97.88% (10,719,359B) (heap allocation functions) malloc/new/new[], --alloc-fns, etc. ->91.19% (9,987,510B) 0x953317: __zend_malloc (zend_alloc.c:2904) | ->76.79% (8,409,934B) 0x9525C8: _emalloc (zend_alloc.c:2494) | | ->73.05% (8,000,688B) 0x89116A: _mysqlnd_emalloc (mysqlnd_alloc.c:99) | | | ->72.90% (7,984,000B) 0x8D3CDD: mysqlnd_arena_alloc (mysqlnd_block_alloc.c:66) | | | | ->72.90% (7,984,000B) 0x8D430D: mysqlnd_mempool_get_chunk (mysqlnd_block_alloc.c:148) | | | | ->72.90% (7,984,000B) 0x8A119A: php_mysqlnd_read_row_ex (mysqlnd_wireprotocol.c:1384) | | | | ->72.90% (7,984,000B) 0x8A358B: php_mysqlnd_rowp_read (mysqlnd_wireprotocol.c:1724) | | | | ->72.90% (7,984,000B) 0x8BF0CE: mysqlnd_mysqlnd_result_unbuffered_fetch_row_c_pub (mysqlnd_result.c:677) | | | | ->72.90% (7,984,000B) 0x8C5755: mysqlnd_mysqlnd_res_fetch_row_c_pub (mysqlnd_result.c:1757) | | | | ->72.90% (7,984,000B) 0x65C177: pdo_mysql_stmt_fetch (mysql_statement.c:660) | | | | ->72.90% (7,984,000B) 0x64853F: do_fetch_common (pdo_stmt.c:676) | | | | ->72.90% (7,984,000B) 0x648BF1: do_fetch (pdo_stmt.c:831) | | | | ->72.90% (7,984,000B) 0x64F1BF: pdo_stmt_iter_move_forwards (pdo_stmt.c:2447) | | | | ->72.90% (7,984,000B) 0xA214AA: ZEND_FE_FETCH_R_SPEC_VAR_HANDLER (zend_vm_execute.h:21484) | | | | ->72.90% (7,984,000B) 0xA624CD: execute_ex (zend_vm_execute.h:57707) | | | | ->72.90% (7,984,000B) 0xA656DA: zend_execute (zend_vm_execute.h:60833) | | | | ->72.90% (7,984,000B) 0x98AE19: zend_execute_scripts (zend.c:1568) | | | | ->72.90% (7,984,000B) 0x8F0BE1: php_execute_script (main.c:2630) | | | | ->72.90% (7,984,000B) 0xA6842F: do_cli (php_cli.c:997) | | | | ->72.90% (7,984,000B) 0xA695A5: main (php_cli.c:1389) Presumably freeing chunks from the arena doesn't work properly anymore. Previous Comments: ------------------------------------------------------------------------ [2019-01-02 10:58:17] nikic@php.net Test script including (slow) table generation: <?php ini_set('memory_limit', '8M'); $pdo = new PDO('mysql:host=127.0.0.1;dbname=test', 'php-test', 'password'); $pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION); $pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false); $pdo->exec("DROP TABLE IF EXISTS large_table"); $pdo->exec("CREATE TABLE large_table (id int)"); for ($j = 0; $j < 100; $j++) { $query = "INSERT INTO large_table (id) VALUES "; for ($i = 0; $i < 10000; $i++) { $query .= "($i), "; } $query .= "($i)"; $pdo->exec($query); } $result = $pdo->query("SELECT * FROM large_table"); foreach ($result as $i => $row) if ($i % 10000 === 0) printf("%10s %10s\n", $i, memory_get_usage()); I'm also getting a Warning: Packets out of order. Expected 231 received 48. Packet size=3422212 in Unknown on line 0 after the memory limit error. ------------------------------------------------------------------------ [2018-12-28 21:06:31] cweiske@php.net This breaks Roundcube webmail, see https://bugs.archlinux.org/task/61139. Happens here with 7.3.0-2 (Debian 9) ------------------------------------------------------------------------ [2018-12-17 09:16:01] sjon at hortensius dot net Description: ------------ commit f1f1f6 broke unbuffered queries in mysqlnd; there is a memory leak causing php to run out of memory Test script: --------------- <?php ini_set('memory_limit', '8M'); $pdo = new PDO('mysql:host=*;dbname=*', '*', '*'); $pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION); $pdo->setAttribute(PDO::MYSQL_ATTR_USE_BUFFERED_QUERY, false); $result = $pdo->query("SELECT * FROM large_table"); foreach ($result as $i => $row) if ($i % 10000 === 0) printf("%10s %10s\n", $i, memory_get_usage()); Expected result: ---------------- reasonable constant memory usage, eg: 0 427648 10000 427936 20000 427968 30000 427968 40000 427936 Actual result: -------------- 0 427296 10000 2000288 20000 3622336 30000 5260736 40000 6915488 Fatal error: Allowed memory size of 8388608 bytes exhausted (tried to allocate 16384 bytes) in /php on line 10 ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=77308&edit=1