Bug #77289 [Com]: php-fpm workers are segfaulting intermittently

Edit report at https://bugs.php.net/bug.php?id=77289&edit=1 ID: 77289 Comment by: lauri dot kentta at gmail dot com Reported by: bugs dot php dot net at mundpropaganda dot net Summary: php-fpm workers are segfaulting intermittently Status: Open Type: Bug Package: opcache Operating System: Linux 4.14.82 PHP Version: 7.3.0 Block user comment: N Private report: N New Comment: I spent a whole day on this. What I've found out so far: The memory manager resets between requests, reusing the same memory blocks for the next request, but some part of PHP has a "persistent" memory block which, after being freed, causes the same block to appear twice in the free_slots list. At this point, the linked list heap->free_slot[bin_num] (slot [5] in my code) ends up with a duplicate entry, which obviously leads to the same memory slot being allocated multiple times. After the second allocation, the slot already contains user-defined data but is still used as the next pointer in the linked list, which causes the occassional SEGFAULT. This could even allow remote code execution, if ASLR didn't make it so difficult. Currently mysqlnd is the primary suspect considering the following backtraces. (Note that numbers in zend_alloc.c are a bit off because of my debugging functions.) Backtraces: Allocating 0x7fc2a3277030, which will then live past zend_mm_shutdown. Thread 2.1 "php-fpm" hit Breakpoint 1, my_break (ptr=0x7fc2a3277030) at Zend/zend_alloc.c:347 #0 my_break (ptr=0x7fc2a3277030) at Zend/zend_alloc.c:347 #1 0x000055ad3ddbbc97 in my_alloc_debug (ptr=140473937653808) at Zend/zend_alloc.c:355 #2 0x000055ad3ddc7fe2 in zend_mm_alloc_small (bin_num=5, size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1360 #3 zend_mm_alloc_heap (size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1442 #4 _emalloc (size=41) at Zend/zend_alloc.c:2618 #5 0x000055ad3dd51d97 in _mysqlnd_emalloc (size=41) at ext/mysqlnd/mysqlnd_alloc.c:99 #6 0x000055ad3dd597e8 in php_mysqlnd_rset_header_read (conn=0x55ad3eb59650, _packet=0x7fff7b65a640) at ext/mysqlnd/mysqlnd_wireprotocol.c:1106 #7 0x000055ad3dd67461 in mysqlnd_query_read_result_set_header (conn=0x55ad3eb59650, s=0x0) at ext/mysqlnd/mysqlnd_result.c:392 #8 0x000055ad3dd71f9b in mysqlnd_com_reap_result_run (cmd=0x7fff7b65a960) at ext/mysqlnd/mysqlnd_commands.c:719 #9 0x000055ad3dd72020 in mysqlnd_com_reap_result_run_command (args=0x7fff7b65a9a0) at ext/mysqlnd/mysqlnd_commands.c:736 #10 0x000055ad3dd73d3c in _mysqlnd_run_command (command=COM_REAP_RESULT) at ext/mysqlnd/mysqlnd_commands.c:1345 #11 0x000055ad3dd4bb77 in mysqlnd_mysqlnd_conn_data_reap_query_pub (conn=0x55ad3eb59650, type=MYSQLND_REAP_RESULT_IMPLICIT) at ext/mysqlnd/mysqlnd_connection.c:904 #12 0x000055ad3dd4b776 in mysqlnd_mysqlnd_conn_data_query_pub (conn=0x55ad3eb59650, query=0x7fc2a3279118 "UPDATE tmp_database.tmp_table SET x = x", query_len=39) at ext/mysqlnd/mysqlnd_connection.c:851 #13 0x000055ad3dc9163f in mysql_handle_doer (dbh=0x55ad3eb52e80, sql=<optimized out>, sql_len=<optimized out>) at ext/pdo_mysql/mysql_driver.c:259 #14 0x000055ad3dc86e85 in zim_PDO_exec (execute_data=<optimized out>, return_value=0x7fff7b65ab80) at ext/pdo/pdo_dbh.c:923 #15 0x000055ad3de7b154 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER (execute_data=0x7fc2a321d020) at Zend/zend_vm_execute.h:983 #16 0x000055ad3de3bedc in execute_ex (ex=<optimized out>) at Zend/zend_vm_execute.h:55012 #17 0x000055ad3de7b413 in zend_execute (op_array=op_array@entry=0x7fc2a32660e0, return_value=return_value@entry=0x0) at Zend/zend_vm_execute.h:60595 #18 0x000055ad3ddedecc in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at Zend/zend.c:1615 #19 0x000055ad3dd83f86 in php_execute_script (primary_file=<optimized out>) at main/main.c:2633 #20 0x000055ad3de88634 in main (argc=<optimized out>, argv=<optimized out>) at sapi/fpm/fpm/fpm_main.c:1945 Continuing. Allocating 0x7fc2a3277090, which will then live past zend_mm_shutdown. Thread 2.1 "php-fpm" hit Breakpoint 1, my_break (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:347 #0 my_break (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:347 #1 0x000055ad3ddbbcc1 in my_alloc_debug (ptr=140473937653904) at Zend/zend_alloc.c:358 #2 0x000055ad3ddc7fe2 in zend_mm_alloc_small (bin_num=5, size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1360 #3 zend_mm_alloc_heap (size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1442 #4 _emalloc (size=41) at Zend/zend_alloc.c:2618 #5 0x000055ad3dd53c03 in _mysqlnd_pestrndup (ptr=0x7fc2a3277030 "Rows matched: 0 Changed: 0 Warnings: 0", length=40, persistent=0 '\000') at ext/mysqlnd/mysqlnd_alloc.c:589 #6 0x000055ad3dd679b4 in mysqlnd_query_read_result_set_header (conn=0x55ad3eb59650, s=0x0) at ext/mysqlnd/mysqlnd_result.c:442 #7 0x000055ad3dd71f9b in mysqlnd_com_reap_result_run (cmd=0x7fff7b65a960) at ext/mysqlnd/mysqlnd_commands.c:719 #8 0x000055ad3dd72020 in mysqlnd_com_reap_result_run_command (args=0x7fff7b65a9a0) at ext/mysqlnd/mysqlnd_commands.c:736 #9 0x000055ad3dd73d3c in _mysqlnd_run_command (command=COM_REAP_RESULT) at ext/mysqlnd/mysqlnd_commands.c:1345 #10 0x000055ad3dd4bb77 in mysqlnd_mysqlnd_conn_data_reap_query_pub (conn=0x55ad3eb59650, type=MYSQLND_REAP_RESULT_IMPLICIT) at ext/mysqlnd/mysqlnd_connection.c:904 #11 0x000055ad3dd4b776 in mysqlnd_mysqlnd_conn_data_query_pub (conn=0x55ad3eb59650, query=0x7fc2a3279118 "UPDATE tmp_database.tmp_table SET x = x", query_len=39) at ext/mysqlnd/mysqlnd_connection.c:851 #12 0x000055ad3dc9163f in mysql_handle_doer (dbh=0x55ad3eb52e80, sql=<optimized out>, sql_len=<optimized out>) at ext/pdo_mysql/mysql_driver.c:259 #13 0x000055ad3dc86e85 in zim_PDO_exec (execute_data=<optimized out>, return_value=0x7fff7b65ab80) at ext/pdo/pdo_dbh.c:923 #14 0x000055ad3de7b154 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER (execute_data=0x7fc2a321d020) at Zend/zend_vm_execute.h:983 #15 0x000055ad3de3bedc in execute_ex (ex=<optimized out>) at Zend/zend_vm_execute.h:55012 #16 0x000055ad3de7b413 in zend_execute (op_array=op_array@entry=0x7fc2a32660e0, return_value=return_value@entry=0x0) at Zend/zend_vm_execute.h:60595 #17 0x000055ad3ddedecc in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at Zend/zend.c:1615 #18 0x000055ad3dd83f86 in php_execute_script (primary_file=<optimized out>) at main/main.c:2633 #19 0x000055ad3de88634 in main (argc=<optimized out>, argv=<optimized out>) at sapi/fpm/fpm/fpm_main.c:1945 Continuing. zend_mm_shutdown is called here. Freeing 0x7fc2a3277090, which now becomes a duplicate entry. Thread 2.1 "php-fpm" hit Breakpoint 1, my_break (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:347 #0 my_break (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:347 #1 0x000055ad3ddbbd2a in heap_free_slot_5_init (p=0x7fc2a3277090) at Zend/zend_alloc.c:369 #2 0x000055ad3ddc81e9 in zend_mm_free_small (bin_num=5, ptr=0x7fc2a3277090, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1390 #3 zend_mm_free_heap (ptr=0x7fc2a3277090, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1486 #4 _efree (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:2633 #5 0x000055ad3dd52cc1 in _mysqlnd_efree (ptr=0x7fc2a3277090) at ext/mysqlnd/mysqlnd_alloc.c:349 #6 0x000055ad3dd5e9fb in mysqlnd_mysqlnd_protocol_send_command_handle_OK_pub (payload_decoder_factory=0x55ad3eb5b7a0, error_info=0x55ad3eb59770, upsert_status=0x55ad3eb59738, ignore_upsert_status=1 '\001', last_message=0x55ad3eb59758) at ext/mysqlnd/mysqlnd_wireprotocol.c:2576 #7 0x000055ad3dd5ee76 in mysqlnd_mysqlnd_protocol_send_command_handle_response_pub (payload_decoder_factory=0x55ad3eb5b7a0, ok_packet=PROT_OK_PACKET, silent=1 '\001', command=COM_PING, ignore_upsert_status=1 '\001', error_info=0x55ad3eb59770, upsert_status=0x55ad3eb59738, last_message=0x55ad3eb59758) at ext/mysqlnd/mysqlnd_wireprotocol.c:2656 #8 0x000055ad3dd70dac in mysqlnd_com_ping_run (cmd=0x7fff7b65a780) at ext/mysqlnd/mysqlnd_commands.c:244 #9 0x000055ad3dd70e58 in mysqlnd_com_ping_run_command (args=0x7fff7b65a7c0) at ext/mysqlnd/mysqlnd_commands.c:268 #10 0x000055ad3dd73c86 in _mysqlnd_run_command (command=COM_PING) at ext/mysqlnd/mysqlnd_commands.c:1324 #11 0x000055ad3dd4c2fc in mysqlnd_mysqlnd_conn_data_ping_pub (conn=0x55ad3eb59650) at ext/mysqlnd/mysqlnd_connection.c:1090 #12 0x000055ad3dc9019e in pdo_mysql_check_liveness (dbh=<optimized out>) at ext/pdo_mysql/mysql_driver.c:505 #13 0x000055ad3dc87e00 in zim_PDO_dbh_constructor (execute_data=0x7fc2a321d100, return_value=<optimized out>) at ext/pdo/pdo_dbh.c:299 #14 0x000055ad3de7b154 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER (execute_data=0x7fc2a321d020) at Zend/zend_vm_execute.h:983 #15 0x000055ad3de3bedc in execute_ex (ex=<optimized out>) at Zend/zend_vm_execute.h:55012 #16 0x000055ad3de7b413 in zend_execute (op_array=op_array@entry=0x7fc2a32660e0, return_value=return_value@entry=0x0) at Zend/zend_vm_execute.h:60595 #17 0x000055ad3ddedecc in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at Zend/zend.c:1615 #18 0x000055ad3dd83f86 in php_execute_script (primary_file=<optimized out>) at main/main.c:2633 #19 0x000055ad3de88634 in main (argc=<optimized out>, argv=<optimized out>) at sapi/fpm/fpm/fpm_main.c:1945 Freeing 0x7fc2a3277030, which now becomes a duplicate entry. Thread 2.1 "php-fpm" hit Breakpoint 1, my_break (ptr=0x7fc2a3277030) at Zend/zend_alloc.c:347 #0 my_break (ptr=0x7fc2a3277030) at Zend/zend_alloc.c:347 #1 0x000055ad3ddbbd2a in heap_free_slot_5_init (p=0x7fc2a3277030) at Zend/zend_alloc.c:369 #2 0x000055ad3ddc7fa5 in zend_mm_alloc_small (bin_num=5, size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1354 #3 zend_mm_alloc_heap (size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1442 #4 _emalloc (size=41) at Zend/zend_alloc.c:2618 #5 0x000055ad3dd51d97 in _mysqlnd_emalloc (size=41) at ext/mysqlnd/mysqlnd_alloc.c:99 #6 0x000055ad3dd597e8 in php_mysqlnd_rset_header_read (conn=0x55ad3eb59650, _packet=0x7fff7b65a640) at ext/mysqlnd/mysqlnd_wireprotocol.c:1106 #7 0x000055ad3dd67461 in mysqlnd_query_read_result_set_header (conn=0x55ad3eb59650, s=0x0) at ext/mysqlnd/mysqlnd_result.c:392 #8 0x000055ad3dd71f9b in mysqlnd_com_reap_result_run (cmd=0x7fff7b65a960) at ext/mysqlnd/mysqlnd_commands.c:719 #9 0x000055ad3dd72020 in mysqlnd_com_reap_result_run_command (args=0x7fff7b65a9a0) at ext/mysqlnd/mysqlnd_commands.c:736 #10 0x000055ad3dd73d3c in _mysqlnd_run_command (command=COM_REAP_RESULT) at ext/mysqlnd/mysqlnd_commands.c:1345 #11 0x000055ad3dd4bb77 in mysqlnd_mysqlnd_conn_data_reap_query_pub (conn=0x55ad3eb59650, type=MYSQLND_REAP_RESULT_IMPLICIT) at ext/mysqlnd/mysqlnd_connection.c:904 #12 0x000055ad3dd4b776 in mysqlnd_mysqlnd_conn_data_query_pub (conn=0x55ad3eb59650, query=0x7fc2a3279118 "UPDATE tmp_database.tmp_table SET x = x", query_len=39) at ext/mysqlnd/mysqlnd_connection.c:851 #13 0x000055ad3dc9163f in mysql_handle_doer (dbh=0x55ad3eb52e80, sql=<optimized out>, sql_len=<optimized out>) at ext/pdo_mysql/mysql_driver.c:259 #14 0x000055ad3dc86e85 in zim_PDO_exec (execute_data=<optimized out>, return_value=0x7fff7b65ab80) at ext/pdo/pdo_dbh.c:923 #15 0x000055ad3de7b154 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER (execute_data=0x7fc2a321d020) at Zend/zend_vm_execute.h:983 #16 0x000055ad3de3bedc in execute_ex (ex=<optimized out>) at Zend/zend_vm_execute.h:55012 #17 0x000055ad3de7b413 in zend_execute (op_array=op_array@entry=0x7fc2a32660e0, return_value=return_value@entry=0x0) at Zend/zend_vm_execute.h:60595 #18 0x000055ad3ddedecc in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at Zend/zend.c:1615 #19 0x000055ad3dd83f86 in php_execute_script (primary_file=<optimized out>) at main/main.c:2633 #20 0x000055ad3de88634 in main (argc=<optimized out>, argv=<optimized out>) at sapi/fpm/fpm/fpm_main.c:1945 Previous Comments: ------------------------------------------------------------------------ [2019-01-06 00:33:38] lauri dot kentta at gmail dot com Sometimes php-fpm child segfaults in zend_mm_alloc_small or calls exit(1) in zend_mm_panic. I've managed to cut it down to this weird test case: <?php // Run with FastCGI 100 times in a row. Look at php-fpm log. $pdo = new PDO("mysql:host=localhost", "root", "", array( PDO::ATTR_PERSISTENT => true, PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION, )); $pdo->exec("DROP DATABASE IF EXISTS tmp_database"); $pdo->exec("CREATE DATABASE tmp_database"); $pdo->exec("DROP TABLE IF EXISTS tmp_database.tmp_table"); $pdo->exec("CREATE TEMPORARY TABLE tmp_database.tmp_table (x INT)"); $pdo->exec("UPDATE tmp_database.tmp_table SET x = x"); ?> This triggers the bug in around 10 % of runs. Note that this is visible only in the php-fpm console ("child ... exited with code 1"), not in the script itself. Disabling opcache makes the bug actually easier to reproduce. In 7.3.0 the bug was also triggered by a slightly more complex test case containing autoload and eval (and MySQL UPDATE), and in that case, string lengths (like the length of the autoloaded class name) affected whether the bug was triggered or not. Also, my test case had originally a few hundred lines (real-life code with real-life input), and even changing some unused code or changing the length of some unused string literals (especially around 16 chars) would change the frequency of the bug, sometimes even make it disappear. That's why I'm tempted to blame either strings or memory management in general. However, I couldn't reproduce this at all without MySQL, so it could still be a MySQL-related bug. I even bisected this to commit 4d5330fb, but reverting it didn't fix the problem. ------------------------------------------------------------------------ [2019-01-02 11:27:36] nikic@php.net Assuming this only happens with opcache (based on the package), the first two things to try would be: * Set opcache.optimization_level=0 and see if this resolves the issue. * Set opcache.protect_memory=1 and see if you get reproducible segfaults. A backtrace from one of those should pinpoint the location. I'd also recommend testing with a current snapshot of PHP 7.3, as a number of opcache-related bugs have already been fixed, but not released yet. ------------------------------------------------------------------------ [2018-12-27 09:43:10] lauri dot kentta at gmail dot com I get these crashes with my custom site. It's not related to some specific path or time. Sometimes it's 1.8 seconds from start, sometimes 6 hours, after 1500 requests on average. I can't yet reproduce it reliably. I suspect that something bad is happening with strings or memory allocation. A bit before the crash, I often see ”child [PID] exited with code 1”. And sometimes, instead of a crash, my site starts producing other fatal errors with clearly corrupted code. ”Call to undefined method ClassA::ClassB::ConstInClassB()” where actually ClassA::foo() is called, or an SQL error of a missing table dbname.FOO, where FOO occurs in many other contexts but certainly not in static::$db_table. Could some PHP core developer guess how this kind of corruption could happen? Maybe reviewing related changes between 7.2.13 and 7.3.0 could then be useful. (What is the best git command for that, considering the way PHP branches are forked?) ------------------------------------------------------------------------ [2018-12-18 15:43:41] bugs dot php dot net at mundpropaganda dot net Related To: Bug #77312 ------------------------------------------------------------------------ [2018-12-12 16:10:27] bugs dot php dot net at mundpropaganda dot net Description: ------------ Hey there, after upgrading from PHP 7.2.13 to to 7.3.0 I noticed php-fpm workers frequently crashing while serving WordPress. Downgrading back to 7.2.13 fixed all issues. The crashes seem to happen after I empty the cache of the wp-supercache plugin and access a previously uncached page of the website, although they are rather intermittent; sometimes occurring immediately, sometimes I have to click around a bit for it to trigger and other times it works just fine. I was able to dump the core of a worker process when it happened: Core was generated by php-fpm: pool amm-php73. Program terminated with signal SIGSEGV, Segmentation fault. #0 zend_mm_alloc_small (bin_num=4, size=40, heap=0x7f9fea600040) at /home/krist/tmp/php-7.3.0/Zend/zend_alloc.c:1287 #1 zend_mm_alloc_heap (size=40, heap=0x7f9fea600040) at /home/krist/tmp/php-7.3.0/Zend/zend_alloc.c:1358 #2 _emalloc (size=size@entry=40) at /home/krist/tmp/php-7.3.0/Zend/zend_alloc.c:2498 #3 0x000055e2dc8d84aa in zend_string_alloc (persistent=0, len=8) at /home/krist/tmp/php-7.3.0/Zend/zend_string.h:155 #4 zend_string_init (persistent=0, len=8, str=<optimized out>) at /home/krist/tmp/php-7.3.0/Zend/zend_string.h:155 #5 lex_scan (zendlval=zendlval@entry=0x7ffcbae3dbc0, elem=0x7ffcbae3dc48) at Zend/zend_language_scanner.l:2760 #6 0x000055e2dc8ebe4a in zendlex (elem=elem@entry=0x7ffcbae3dc48) at /home/krist/tmp/php-7.3.0/Zend/zend_compile.c:1693 #7 0x000055e2dc8d172e in zendparse () at /home/krist/tmp/php-7.3.0/Zend/zend_language_parser.c:4211 #8 0x000055e2dc8d3c2a in zend_compile (type=type@entry=2) at Zend/zend_language_scanner.l:586 #9 0x000055e2dc8d540a in compile_file (file_handle=0x7ffcbae3ebc0, type=2) at Zend/zend_language_scanner.l:636 #10 0x000055e2dc7ada72 in phar_compile_file (file_handle=0x7ffcbae3ebc0, type=2) at /home/krist/tmp/php-7.3.0/ext/phar/phar.c:3344 #11 0x00007f9fea8a106c in opcache_compile_file (file_handle=0x7ffcbae3ebc0, type=2, op_array_p=0x7ffcbae3eaa8, key=<optimized out>) at /home/krist/tmp/php-7.3.0/ext/opcache/ZendAccelerator.c:1750 #12 0x00007f9fea8a332f in persistent_compile_file (type=2, file_handle=0x7ffcbae3ebc0) at /home/krist/tmp/php-7.3.0/ext/opcache/ZendAccelerator.c:2095 #13 persistent_compile_file (file_handle=0x7ffcbae3ebc0, type=2) at /home/krist/tmp/php-7.3.0/ext/opcache/ZendAccelerator.c:1889 #14 0x000055e2dc8d54b2 in compile_filename (type=type@entry=2, filename=filename@entry=0x7f9fea61da40) at Zend/zend_language_scanner.l:661 #15 0x000055e2dc94dc65 in zend_include_or_eval (inc_filename=inc_filename@entry=0x7f9fea61da40, type=2) at /home/krist/tmp/php-7.3.0/Zend/zend_execute.c:3182 #16 0x000055e2dc983d10 in ZEND_INCLUDE_OR_EVAL_SPEC_TMPVAR_HANDLER () at /home/krist/tmp/php-7.3.0/Zend/zend_vm_execute.h:12697 #17 0x000055e2dc989f0f in execute_ex (ex=0x861e00) at /home/krist/tmp/php-7.3.0/Zend/zend_vm_execute.h:56832 #18 0x000055e2dc990702 in zend_execute (op_array=op_array@entry=0x7f9fe339f958, return_value=0x0, return_value@entry=0x7f9fea61d9a0) at /home/krist/tmp/php-7.3.0/Zend/zend_vm_execute.h:60834 #19 0x000055e2dc90ad83 in zend_execute_scripts (type=type@entry=8, retval=0x7f9fea61d9a0, retval@entry=0x0, file_count=file_count@entry=3) at /home/krist/tmp/php-7.3.0/Zend/zend.c:1568 #20 0x000055e2dc8ac228 in php_execute_script (primary_file=<optimized out>) at /home/krist/tmp/php-7.3.0/main/main.c:2630 #21 0x000055e2dc69f290 in main (argc=<optimized out>, argv=<optimized out>) at /home/krist/tmp/php-7.3.0/sapi/fpm/fpm/fpm_main.c:1950 Let me know if there's anything I might do or provide to help tracing this bug. – Christian ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=77289&edit=1