Bug #77289 [Opn]: php-fpm workers are segfaulting intermittently

Edit report at https://bugs.php.net/bug.php?id=77289&edit=1 ID: 77289 Updated by: nikic@php.net Reported by: bugs dot php dot net at mundpropaganda dot net Summary: php-fpm workers are segfaulting intermittently Status: Open Type: Bug Package: opcache Operating System: Linux 4.14.82 PHP Version: 7.3.0 -Assigned To: +Assigned To: dmitry Block user comment: N Private report: N New Comment: I think the problem is that the mysqlnd connection has end_psession and restart_psession handlers that are supposed to be called at the end/start of a request respectively. If end_psession were called, it would free the last_message and NULL it. However, it seems that mysqli only calls restart_psession (which also nulls last_message, so probably prevents the worst) and PDO doesn't call either (resulting in what we see here). @dmitry: What do you think about this? I think reverting to use persistent flag for last_message is probably the most pragmatic thing to do here. (There might be ABI concerns though...) Previous Comments: ------------------------------------------------------------------------ [2019-01-08 09:37:38] nikic@php.net last_message was changed in https://github.com/php/php-src/commit/a7305eb539596e175bd6c3ae9a20953358c5d677 to be allocated by Zend MM, with the comment that it's not supposed to be used in the next request. I'm not sure if the attached patch is right, or if some use of last_message (or lack of reset somewhere) is at fault here. ------------------------------------------------------------------------ [2019-01-07 19:29:04] lauri dot kentta at gmail dot com Related To: Bug #77312 ------------------------------------------------------------------------ [2019-01-06 21:18:16] bugs dot php dot net at mundpropaganda dot net Hey Lauri, OP here. Just want to say wow and thanks… big time! Your work is greatly appreciated! The other opcache bug I encountered seems already fixed in 7.3.1RC1. Looking fwd to the next patch releases! ------------------------------------------------------------------------ [2019-01-06 20:07:29] lauri dot kentta at gmail dot com Oops, wrong backtrace in the last piece. However, the first one led to the right place, or so it seems. I'll send a patch. ------------------------------------------------------------------------ [2019-01-06 19:42:11] lauri dot kentta at gmail dot com I spent a whole day on this. What I've found out so far: The memory manager resets between requests, reusing the same memory blocks for the next request, but some part of PHP has a "persistent" memory block which, after being freed, causes the same block to appear twice in the free_slots list. At this point, the linked list heap->free_slot[bin_num] (slot [5] in my code) ends up with a duplicate entry, which obviously leads to the same memory slot being allocated multiple times. After the second allocation, the slot already contains user-defined data but is still used as the next pointer in the linked list, which causes the occassional SEGFAULT. This could even allow remote code execution, if ASLR didn't make it so difficult. Currently mysqlnd is the primary suspect considering the following backtraces. (Note that numbers in zend_alloc.c are a bit off because of my debugging functions.) Backtraces: Allocating 0x7fc2a3277030, which will then live past zend_mm_shutdown. Thread 2.1 "php-fpm" hit Breakpoint 1, my_break (ptr=0x7fc2a3277030) at Zend/zend_alloc.c:347 #0 my_break (ptr=0x7fc2a3277030) at Zend/zend_alloc.c:347 #1 0x000055ad3ddbbc97 in my_alloc_debug (ptr=140473937653808) at Zend/zend_alloc.c:355 #2 0x000055ad3ddc7fe2 in zend_mm_alloc_small (bin_num=5, size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1360 #3 zend_mm_alloc_heap (size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1442 #4 _emalloc (size=41) at Zend/zend_alloc.c:2618 #5 0x000055ad3dd51d97 in _mysqlnd_emalloc (size=41) at ext/mysqlnd/mysqlnd_alloc.c:99 #6 0x000055ad3dd597e8 in php_mysqlnd_rset_header_read (conn=0x55ad3eb59650, _packet=0x7fff7b65a640) at ext/mysqlnd/mysqlnd_wireprotocol.c:1106 #7 0x000055ad3dd67461 in mysqlnd_query_read_result_set_header (conn=0x55ad3eb59650, s=0x0) at ext/mysqlnd/mysqlnd_result.c:392 #8 0x000055ad3dd71f9b in mysqlnd_com_reap_result_run (cmd=0x7fff7b65a960) at ext/mysqlnd/mysqlnd_commands.c:719 #9 0x000055ad3dd72020 in mysqlnd_com_reap_result_run_command (args=0x7fff7b65a9a0) at ext/mysqlnd/mysqlnd_commands.c:736 #10 0x000055ad3dd73d3c in _mysqlnd_run_command (command=COM_REAP_RESULT) at ext/mysqlnd/mysqlnd_commands.c:1345 #11 0x000055ad3dd4bb77 in mysqlnd_mysqlnd_conn_data_reap_query_pub (conn=0x55ad3eb59650, type=MYSQLND_REAP_RESULT_IMPLICIT) at ext/mysqlnd/mysqlnd_connection.c:904 #12 0x000055ad3dd4b776 in mysqlnd_mysqlnd_conn_data_query_pub (conn=0x55ad3eb59650, query=0x7fc2a3279118 "UPDATE tmp_database.tmp_table SET x = x", query_len=39) at ext/mysqlnd/mysqlnd_connection.c:851 #13 0x000055ad3dc9163f in mysql_handle_doer (dbh=0x55ad3eb52e80, sql=<optimized out>, sql_len=<optimized out>) at ext/pdo_mysql/mysql_driver.c:259 #14 0x000055ad3dc86e85 in zim_PDO_exec (execute_data=<optimized out>, return_value=0x7fff7b65ab80) at ext/pdo/pdo_dbh.c:923 #15 0x000055ad3de7b154 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER (execute_data=0x7fc2a321d020) at Zend/zend_vm_execute.h:983 #16 0x000055ad3de3bedc in execute_ex (ex=<optimized out>) at Zend/zend_vm_execute.h:55012 #17 0x000055ad3de7b413 in zend_execute (op_array=op_array@entry=0x7fc2a32660e0, return_value=return_value@entry=0x0) at Zend/zend_vm_execute.h:60595 #18 0x000055ad3ddedecc in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at Zend/zend.c:1615 #19 0x000055ad3dd83f86 in php_execute_script (primary_file=<optimized out>) at main/main.c:2633 #20 0x000055ad3de88634 in main (argc=<optimized out>, argv=<optimized out>) at sapi/fpm/fpm/fpm_main.c:1945 Continuing. Allocating 0x7fc2a3277090, which will then live past zend_mm_shutdown. Thread 2.1 "php-fpm" hit Breakpoint 1, my_break (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:347 #0 my_break (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:347 #1 0x000055ad3ddbbcc1 in my_alloc_debug (ptr=140473937653904) at Zend/zend_alloc.c:358 #2 0x000055ad3ddc7fe2 in zend_mm_alloc_small (bin_num=5, size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1360 #3 zend_mm_alloc_heap (size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1442 #4 _emalloc (size=41) at Zend/zend_alloc.c:2618 #5 0x000055ad3dd53c03 in _mysqlnd_pestrndup (ptr=0x7fc2a3277030 "Rows matched: 0 Changed: 0 Warnings: 0", length=40, persistent=0 '\000') at ext/mysqlnd/mysqlnd_alloc.c:589 #6 0x000055ad3dd679b4 in mysqlnd_query_read_result_set_header (conn=0x55ad3eb59650, s=0x0) at ext/mysqlnd/mysqlnd_result.c:442 #7 0x000055ad3dd71f9b in mysqlnd_com_reap_result_run (cmd=0x7fff7b65a960) at ext/mysqlnd/mysqlnd_commands.c:719 #8 0x000055ad3dd72020 in mysqlnd_com_reap_result_run_command (args=0x7fff7b65a9a0) at ext/mysqlnd/mysqlnd_commands.c:736 #9 0x000055ad3dd73d3c in _mysqlnd_run_command (command=COM_REAP_RESULT) at ext/mysqlnd/mysqlnd_commands.c:1345 #10 0x000055ad3dd4bb77 in mysqlnd_mysqlnd_conn_data_reap_query_pub (conn=0x55ad3eb59650, type=MYSQLND_REAP_RESULT_IMPLICIT) at ext/mysqlnd/mysqlnd_connection.c:904 #11 0x000055ad3dd4b776 in mysqlnd_mysqlnd_conn_data_query_pub (conn=0x55ad3eb59650, query=0x7fc2a3279118 "UPDATE tmp_database.tmp_table SET x = x", query_len=39) at ext/mysqlnd/mysqlnd_connection.c:851 #12 0x000055ad3dc9163f in mysql_handle_doer (dbh=0x55ad3eb52e80, sql=<optimized out>, sql_len=<optimized out>) at ext/pdo_mysql/mysql_driver.c:259 #13 0x000055ad3dc86e85 in zim_PDO_exec (execute_data=<optimized out>, return_value=0x7fff7b65ab80) at ext/pdo/pdo_dbh.c:923 #14 0x000055ad3de7b154 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER (execute_data=0x7fc2a321d020) at Zend/zend_vm_execute.h:983 #15 0x000055ad3de3bedc in execute_ex (ex=<optimized out>) at Zend/zend_vm_execute.h:55012 #16 0x000055ad3de7b413 in zend_execute (op_array=op_array@entry=0x7fc2a32660e0, return_value=return_value@entry=0x0) at Zend/zend_vm_execute.h:60595 #17 0x000055ad3ddedecc in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at Zend/zend.c:1615 #18 0x000055ad3dd83f86 in php_execute_script (primary_file=<optimized out>) at main/main.c:2633 #19 0x000055ad3de88634 in main (argc=<optimized out>, argv=<optimized out>) at sapi/fpm/fpm/fpm_main.c:1945 Continuing. zend_mm_shutdown is called here. Freeing 0x7fc2a3277090, which now becomes a duplicate entry. Thread 2.1 "php-fpm" hit Breakpoint 1, my_break (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:347 #0 my_break (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:347 #1 0x000055ad3ddbbd2a in heap_free_slot_5_init (p=0x7fc2a3277090) at Zend/zend_alloc.c:369 #2 0x000055ad3ddc81e9 in zend_mm_free_small (bin_num=5, ptr=0x7fc2a3277090, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1390 #3 zend_mm_free_heap (ptr=0x7fc2a3277090, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1486 #4 _efree (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:2633 #5 0x000055ad3dd52cc1 in _mysqlnd_efree (ptr=0x7fc2a3277090) at ext/mysqlnd/mysqlnd_alloc.c:349 #6 0x000055ad3dd5e9fb in mysqlnd_mysqlnd_protocol_send_command_handle_OK_pub (payload_decoder_factory=0x55ad3eb5b7a0, error_info=0x55ad3eb59770, upsert_status=0x55ad3eb59738, ignore_upsert_status=1 '\001', last_message=0x55ad3eb59758) at ext/mysqlnd/mysqlnd_wireprotocol.c:2576 #7 0x000055ad3dd5ee76 in mysqlnd_mysqlnd_protocol_send_command_handle_response_pub (payload_decoder_factory=0x55ad3eb5b7a0, ok_packet=PROT_OK_PACKET, silent=1 '\001', command=COM_PING, ignore_upsert_status=1 '\001', error_info=0x55ad3eb59770, upsert_status=0x55ad3eb59738, last_message=0x55ad3eb59758) at ext/mysqlnd/mysqlnd_wireprotocol.c:2656 #8 0x000055ad3dd70dac in mysqlnd_com_ping_run (cmd=0x7fff7b65a780) at ext/mysqlnd/mysqlnd_commands.c:244 #9 0x000055ad3dd70e58 in mysqlnd_com_ping_run_command (args=0x7fff7b65a7c0) at ext/mysqlnd/mysqlnd_commands.c:268 #10 0x000055ad3dd73c86 in _mysqlnd_run_command (command=COM_PING) at ext/mysqlnd/mysqlnd_commands.c:1324 #11 0x000055ad3dd4c2fc in mysqlnd_mysqlnd_conn_data_ping_pub (conn=0x55ad3eb59650) at ext/mysqlnd/mysqlnd_connection.c:1090 #12 0x000055ad3dc9019e in pdo_mysql_check_liveness (dbh=<optimized out>) at ext/pdo_mysql/mysql_driver.c:505 #13 0x000055ad3dc87e00 in zim_PDO_dbh_constructor (execute_data=0x7fc2a321d100, return_value=<optimized out>) at ext/pdo/pdo_dbh.c:299 #14 0x000055ad3de7b154 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER (execute_data=0x7fc2a321d020) at Zend/zend_vm_execute.h:983 #15 0x000055ad3de3bedc in execute_ex (ex=<optimized out>) at Zend/zend_vm_execute.h:55012 #16 0x000055ad3de7b413 in zend_execute (op_array=op_array@entry=0x7fc2a32660e0, return_value=return_value@entry=0x0) at Zend/zend_vm_execute.h:60595 #17 0x000055ad3ddedecc in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at Zend/zend.c:1615 #18 0x000055ad3dd83f86 in php_execute_script (primary_file=<optimized out>) at main/main.c:2633 #19 0x000055ad3de88634 in main (argc=<optimized out>, argv=<optimized out>) at sapi/fpm/fpm/fpm_main.c:1945 Freeing 0x7fc2a3277030, which now becomes a duplicate entry. Thread 2.1 "php-fpm" hit Breakpoint 1, my_break (ptr=0x7fc2a3277030) at Zend/zend_alloc.c:347 #0 my_break (ptr=0x7fc2a3277030) at Zend/zend_alloc.c:347 #1 0x000055ad3ddbbd2a in heap_free_slot_5_init (p=0x7fc2a3277030) at Zend/zend_alloc.c:369 #2 0x000055ad3ddc7fa5 in zend_mm_alloc_small (bin_num=5, size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1354 #3 zend_mm_alloc_heap (size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1442 #4 _emalloc (size=41) at Zend/zend_alloc.c:2618 #5 0x000055ad3dd51d97 in _mysqlnd_emalloc (size=41) at ext/mysqlnd/mysqlnd_alloc.c:99 #6 0x000055ad3dd597e8 in php_mysqlnd_rset_header_read (conn=0x55ad3eb59650, _packet=0x7fff7b65a640) at ext/mysqlnd/mysqlnd_wireprotocol.c:1106 #7 0x000055ad3dd67461 in mysqlnd_query_read_result_set_header (conn=0x55ad3eb59650, s=0x0) at ext/mysqlnd/mysqlnd_result.c:392 #8 0x000055ad3dd71f9b in mysqlnd_com_reap_result_run (cmd=0x7fff7b65a960) at ext/mysqlnd/mysqlnd_commands.c:719 #9 0x000055ad3dd72020 in mysqlnd_com_reap_result_run_command (args=0x7fff7b65a9a0) at ext/mysqlnd/mysqlnd_commands.c:736 #10 0x000055ad3dd73d3c in _mysqlnd_run_command (command=COM_REAP_RESULT) at ext/mysqlnd/mysqlnd_commands.c:1345 #11 0x000055ad3dd4bb77 in mysqlnd_mysqlnd_conn_data_reap_query_pub (conn=0x55ad3eb59650, type=MYSQLND_REAP_RESULT_IMPLICIT) at ext/mysqlnd/mysqlnd_connection.c:904 #12 0x000055ad3dd4b776 in mysqlnd_mysqlnd_conn_data_query_pub (conn=0x55ad3eb59650, query=0x7fc2a3279118 "UPDATE tmp_database.tmp_table SET x = x", query_len=39) at ext/mysqlnd/mysqlnd_connection.c:851 #13 0x000055ad3dc9163f in mysql_handle_doer (dbh=0x55ad3eb52e80, sql=<optimized out>, sql_len=<optimized out>) at ext/pdo_mysql/mysql_driver.c:259 #14 0x000055ad3dc86e85 in zim_PDO_exec (execute_data=<optimized out>, return_value=0x7fff7b65ab80) at ext/pdo/pdo_dbh.c:923 #15 0x000055ad3de7b154 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER (execute_data=0x7fc2a321d020) at Zend/zend_vm_execute.h:983 #16 0x000055ad3de3bedc in execute_ex (ex=<optimized out>) at Zend/zend_vm_execute.h:55012 #17 0x000055ad3de7b413 in zend_execute (op_array=op_array@entry=0x7fc2a32660e0, return_value=return_value@entry=0x0) at Zend/zend_vm_execute.h:60595 #18 0x000055ad3ddedecc in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at Zend/zend.c:1615 #19 0x000055ad3dd83f86 in php_execute_script (primary_file=<optimized out>) at main/main.c:2633 #20 0x000055ad3de88634 in main (argc=<optimized out>, argv=<optimized out>) at sapi/fpm/fpm/fpm_main.c:1945 ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=77289 -- Edit this bug report at https://bugs.php.net/bug.php?id=77289&edit=1