Edit report at https://bugs.php.net/bug.php?id=77289&edit=1
ID: 77289
Comment by: lauri dot kentta at gmail dot com
Reported by: bugs dot php dot net at mundpropaganda dot net
Summary: php-fpm workers are segfaulting intermittently
Status: Open
Type: Bug
Package: opcache
Operating System: Linux 4.14.82
PHP Version: 7.3.0
Block user comment: N
Private report: N
New Comment:
Oops, wrong backtrace in the last piece. However, the first one led to the right place, or so it
seems. I'll send a patch.
Previous Comments:
------------------------------------------------------------------------
[2019-01-06 19:42:11] lauri dot kentta at gmail dot com
I spent a whole day on this. What I've found out so far:
The memory manager resets between requests, reusing the same memory blocks for the next request, but
some part of PHP has a "persistent" memory block which, after being freed, causes the same
block to appear twice in the free_slots list.
At this point, the linked list heap->free_slot[bin_num] (slot [5] in my code) ends up with a
duplicate entry, which obviously leads to the same memory slot being allocated multiple times. After
the second allocation, the slot already contains user-defined data but is still used as the next
pointer in the linked list, which causes the occassional SEGFAULT. This could even allow remote code
execution, if ASLR didn't make it so difficult.
Currently mysqlnd is the primary suspect considering the following backtraces. (Note that numbers in
zend_alloc.c are a bit off because of my debugging functions.)
Backtraces:
Allocating 0x7fc2a3277030, which will then live past zend_mm_shutdown.
Thread 2.1 "php-fpm" hit Breakpoint 1, my_break (ptr=0x7fc2a3277030) at
Zend/zend_alloc.c:347
#0 my_break (ptr=0x7fc2a3277030) at Zend/zend_alloc.c:347
#1 0x000055ad3ddbbc97 in my_alloc_debug (ptr=140473937653808) at Zend/zend_alloc.c:355
#2 0x000055ad3ddc7fe2 in zend_mm_alloc_small (bin_num=5, size=41, heap=0x7fc2a3200040) at
Zend/zend_alloc.c:1360
#3 zend_mm_alloc_heap (size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1442
#4 _emalloc (size=41) at Zend/zend_alloc.c:2618
#5 0x000055ad3dd51d97 in _mysqlnd_emalloc (size=41) at ext/mysqlnd/mysqlnd_alloc.c:99
#6 0x000055ad3dd597e8 in php_mysqlnd_rset_header_read (conn=0x55ad3eb59650, _packet=0x7fff7b65a640)
at ext/mysqlnd/mysqlnd_wireprotocol.c:1106
#7 0x000055ad3dd67461 in mysqlnd_query_read_result_set_header (conn=0x55ad3eb59650, s=0x0) at
ext/mysqlnd/mysqlnd_result.c:392
#8 0x000055ad3dd71f9b in mysqlnd_com_reap_result_run (cmd=0x7fff7b65a960) at
ext/mysqlnd/mysqlnd_commands.c:719
#9 0x000055ad3dd72020 in mysqlnd_com_reap_result_run_command (args=0x7fff7b65a9a0) at
ext/mysqlnd/mysqlnd_commands.c:736
#10 0x000055ad3dd73d3c in _mysqlnd_run_command (command=COM_REAP_RESULT) at
ext/mysqlnd/mysqlnd_commands.c:1345
#11 0x000055ad3dd4bb77 in mysqlnd_mysqlnd_conn_data_reap_query_pub (conn=0x55ad3eb59650,
type=MYSQLND_REAP_RESULT_IMPLICIT) at ext/mysqlnd/mysqlnd_connection.c:904
#12 0x000055ad3dd4b776 in mysqlnd_mysqlnd_conn_data_query_pub (conn=0x55ad3eb59650,
query=0x7fc2a3279118 "UPDATE tmp_database.tmp_table SET x = x", query_len=39) at
ext/mysqlnd/mysqlnd_connection.c:851
#13 0x000055ad3dc9163f in mysql_handle_doer (dbh=0x55ad3eb52e80, sql=<optimized out>,
sql_len=<optimized out>) at ext/pdo_mysql/mysql_driver.c:259
#14 0x000055ad3dc86e85 in zim_PDO_exec (execute_data=<optimized out>,
return_value=0x7fff7b65ab80) at ext/pdo/pdo_dbh.c:923
#15 0x000055ad3de7b154 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER (execute_data=0x7fc2a321d020) at
Zend/zend_vm_execute.h:983
#16 0x000055ad3de3bedc in execute_ex (ex=<optimized out>) at Zend/zend_vm_execute.h:55012
#17 0x000055ad3de7b413 in zend_execute (op_array=op_array@entry=0x7fc2a32660e0,
return_value=return_value@entry=0x0) at Zend/zend_vm_execute.h:60595
#18 0x000055ad3ddedecc in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0,
file_count=file_count@entry=3) at Zend/zend.c:1615
#19 0x000055ad3dd83f86 in php_execute_script (primary_file=<optimized out>) at
main/main.c:2633
#20 0x000055ad3de88634 in main (argc=<optimized out>, argv=<optimized out>) at
sapi/fpm/fpm/fpm_main.c:1945
Continuing.
Allocating 0x7fc2a3277090, which will then live past zend_mm_shutdown.
Thread 2.1 "php-fpm" hit Breakpoint 1, my_break (ptr=0x7fc2a3277090) at
Zend/zend_alloc.c:347
#0 my_break (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:347
#1 0x000055ad3ddbbcc1 in my_alloc_debug (ptr=140473937653904) at Zend/zend_alloc.c:358
#2 0x000055ad3ddc7fe2 in zend_mm_alloc_small (bin_num=5, size=41, heap=0x7fc2a3200040) at
Zend/zend_alloc.c:1360
#3 zend_mm_alloc_heap (size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1442
#4 _emalloc (size=41) at Zend/zend_alloc.c:2618
#5 0x000055ad3dd53c03 in _mysqlnd_pestrndup (ptr=0x7fc2a3277030 "Rows matched: 0 Changed: 0
Warnings: 0", length=40, persistent=0 '\000') at ext/mysqlnd/mysqlnd_alloc.c:589
#6 0x000055ad3dd679b4 in mysqlnd_query_read_result_set_header (conn=0x55ad3eb59650, s=0x0) at
ext/mysqlnd/mysqlnd_result.c:442
#7 0x000055ad3dd71f9b in mysqlnd_com_reap_result_run (cmd=0x7fff7b65a960) at
ext/mysqlnd/mysqlnd_commands.c:719
#8 0x000055ad3dd72020 in mysqlnd_com_reap_result_run_command (args=0x7fff7b65a9a0) at
ext/mysqlnd/mysqlnd_commands.c:736
#9 0x000055ad3dd73d3c in _mysqlnd_run_command (command=COM_REAP_RESULT) at
ext/mysqlnd/mysqlnd_commands.c:1345
#10 0x000055ad3dd4bb77 in mysqlnd_mysqlnd_conn_data_reap_query_pub (conn=0x55ad3eb59650,
type=MYSQLND_REAP_RESULT_IMPLICIT) at ext/mysqlnd/mysqlnd_connection.c:904
#11 0x000055ad3dd4b776 in mysqlnd_mysqlnd_conn_data_query_pub (conn=0x55ad3eb59650,
query=0x7fc2a3279118 "UPDATE tmp_database.tmp_table SET x = x", query_len=39) at
ext/mysqlnd/mysqlnd_connection.c:851
#12 0x000055ad3dc9163f in mysql_handle_doer (dbh=0x55ad3eb52e80, sql=<optimized out>,
sql_len=<optimized out>) at ext/pdo_mysql/mysql_driver.c:259
#13 0x000055ad3dc86e85 in zim_PDO_exec (execute_data=<optimized out>,
return_value=0x7fff7b65ab80) at ext/pdo/pdo_dbh.c:923
#14 0x000055ad3de7b154 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER (execute_data=0x7fc2a321d020) at
Zend/zend_vm_execute.h:983
#15 0x000055ad3de3bedc in execute_ex (ex=<optimized out>) at Zend/zend_vm_execute.h:55012
#16 0x000055ad3de7b413 in zend_execute (op_array=op_array@entry=0x7fc2a32660e0,
return_value=return_value@entry=0x0) at Zend/zend_vm_execute.h:60595
#17 0x000055ad3ddedecc in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0,
file_count=file_count@entry=3) at Zend/zend.c:1615
#18 0x000055ad3dd83f86 in php_execute_script (primary_file=<optimized out>) at
main/main.c:2633
#19 0x000055ad3de88634 in main (argc=<optimized out>, argv=<optimized out>) at
sapi/fpm/fpm/fpm_main.c:1945
Continuing.
zend_mm_shutdown is called here.
Freeing 0x7fc2a3277090, which now becomes a duplicate entry.
Thread 2.1 "php-fpm" hit Breakpoint 1, my_break (ptr=0x7fc2a3277090) at
Zend/zend_alloc.c:347
#0 my_break (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:347
#1 0x000055ad3ddbbd2a in heap_free_slot_5_init (p=0x7fc2a3277090) at Zend/zend_alloc.c:369
#2 0x000055ad3ddc81e9 in zend_mm_free_small (bin_num=5, ptr=0x7fc2a3277090, heap=0x7fc2a3200040) at
Zend/zend_alloc.c:1390
#3 zend_mm_free_heap (ptr=0x7fc2a3277090, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1486
#4 _efree (ptr=0x7fc2a3277090) at Zend/zend_alloc.c:2633
#5 0x000055ad3dd52cc1 in _mysqlnd_efree (ptr=0x7fc2a3277090) at ext/mysqlnd/mysqlnd_alloc.c:349
#6 0x000055ad3dd5e9fb in mysqlnd_mysqlnd_protocol_send_command_handle_OK_pub
(payload_decoder_factory=0x55ad3eb5b7a0, error_info=0x55ad3eb59770, upsert_status=0x55ad3eb59738,
ignore_upsert_status=1 '\001', last_message=0x55ad3eb59758) at
ext/mysqlnd/mysqlnd_wireprotocol.c:2576
#7 0x000055ad3dd5ee76 in mysqlnd_mysqlnd_protocol_send_command_handle_response_pub
(payload_decoder_factory=0x55ad3eb5b7a0, ok_packet=PROT_OK_PACKET, silent=1 '\001',
command=COM_PING, ignore_upsert_status=1 '\001', error_info=0x55ad3eb59770,
upsert_status=0x55ad3eb59738, last_message=0x55ad3eb59758) at
ext/mysqlnd/mysqlnd_wireprotocol.c:2656
#8 0x000055ad3dd70dac in mysqlnd_com_ping_run (cmd=0x7fff7b65a780) at
ext/mysqlnd/mysqlnd_commands.c:244
#9 0x000055ad3dd70e58 in mysqlnd_com_ping_run_command (args=0x7fff7b65a7c0) at
ext/mysqlnd/mysqlnd_commands.c:268
#10 0x000055ad3dd73c86 in _mysqlnd_run_command (command=COM_PING) at
ext/mysqlnd/mysqlnd_commands.c:1324
#11 0x000055ad3dd4c2fc in mysqlnd_mysqlnd_conn_data_ping_pub (conn=0x55ad3eb59650) at
ext/mysqlnd/mysqlnd_connection.c:1090
#12 0x000055ad3dc9019e in pdo_mysql_check_liveness (dbh=<optimized out>) at
ext/pdo_mysql/mysql_driver.c:505
#13 0x000055ad3dc87e00 in zim_PDO_dbh_constructor (execute_data=0x7fc2a321d100,
return_value=<optimized out>) at ext/pdo/pdo_dbh.c:299
#14 0x000055ad3de7b154 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER (execute_data=0x7fc2a321d020) at
Zend/zend_vm_execute.h:983
#15 0x000055ad3de3bedc in execute_ex (ex=<optimized out>) at Zend/zend_vm_execute.h:55012
#16 0x000055ad3de7b413 in zend_execute (op_array=op_array@entry=0x7fc2a32660e0,
return_value=return_value@entry=0x0) at Zend/zend_vm_execute.h:60595
#17 0x000055ad3ddedecc in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0,
file_count=file_count@entry=3) at Zend/zend.c:1615
#18 0x000055ad3dd83f86 in php_execute_script (primary_file=<optimized out>) at
main/main.c:2633
#19 0x000055ad3de88634 in main (argc=<optimized out>, argv=<optimized out>) at
sapi/fpm/fpm/fpm_main.c:1945
Freeing 0x7fc2a3277030, which now becomes a duplicate entry.
Thread 2.1 "php-fpm" hit Breakpoint 1, my_break (ptr=0x7fc2a3277030) at
Zend/zend_alloc.c:347
#0 my_break (ptr=0x7fc2a3277030) at Zend/zend_alloc.c:347
#1 0x000055ad3ddbbd2a in heap_free_slot_5_init (p=0x7fc2a3277030) at Zend/zend_alloc.c:369
#2 0x000055ad3ddc7fa5 in zend_mm_alloc_small (bin_num=5, size=41, heap=0x7fc2a3200040) at
Zend/zend_alloc.c:1354
#3 zend_mm_alloc_heap (size=41, heap=0x7fc2a3200040) at Zend/zend_alloc.c:1442
#4 _emalloc (size=41) at Zend/zend_alloc.c:2618
#5 0x000055ad3dd51d97 in _mysqlnd_emalloc (size=41) at ext/mysqlnd/mysqlnd_alloc.c:99
#6 0x000055ad3dd597e8 in php_mysqlnd_rset_header_read (conn=0x55ad3eb59650, _packet=0x7fff7b65a640)
at ext/mysqlnd/mysqlnd_wireprotocol.c:1106
#7 0x000055ad3dd67461 in mysqlnd_query_read_result_set_header (conn=0x55ad3eb59650, s=0x0) at
ext/mysqlnd/mysqlnd_result.c:392
#8 0x000055ad3dd71f9b in mysqlnd_com_reap_result_run (cmd=0x7fff7b65a960) at
ext/mysqlnd/mysqlnd_commands.c:719
#9 0x000055ad3dd72020 in mysqlnd_com_reap_result_run_command (args=0x7fff7b65a9a0) at
ext/mysqlnd/mysqlnd_commands.c:736
#10 0x000055ad3dd73d3c in _mysqlnd_run_command (command=COM_REAP_RESULT) at
ext/mysqlnd/mysqlnd_commands.c:1345
#11 0x000055ad3dd4bb77 in mysqlnd_mysqlnd_conn_data_reap_query_pub (conn=0x55ad3eb59650,
type=MYSQLND_REAP_RESULT_IMPLICIT) at ext/mysqlnd/mysqlnd_connection.c:904
#12 0x000055ad3dd4b776 in mysqlnd_mysqlnd_conn_data_query_pub (conn=0x55ad3eb59650,
query=0x7fc2a3279118 "UPDATE tmp_database.tmp_table SET x = x", query_len=39) at
ext/mysqlnd/mysqlnd_connection.c:851
#13 0x000055ad3dc9163f in mysql_handle_doer (dbh=0x55ad3eb52e80, sql=<optimized out>,
sql_len=<optimized out>) at ext/pdo_mysql/mysql_driver.c:259
#14 0x000055ad3dc86e85 in zim_PDO_exec (execute_data=<optimized out>,
return_value=0x7fff7b65ab80) at ext/pdo/pdo_dbh.c:923
#15 0x000055ad3de7b154 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER (execute_data=0x7fc2a321d020) at
Zend/zend_vm_execute.h:983
#16 0x000055ad3de3bedc in execute_ex (ex=<optimized out>) at Zend/zend_vm_execute.h:55012
#17 0x000055ad3de7b413 in zend_execute (op_array=op_array@entry=0x7fc2a32660e0,
return_value=return_value@entry=0x0) at Zend/zend_vm_execute.h:60595
#18 0x000055ad3ddedecc in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0,
file_count=file_count@entry=3) at Zend/zend.c:1615
#19 0x000055ad3dd83f86 in php_execute_script (primary_file=<optimized out>) at
main/main.c:2633
#20 0x000055ad3de88634 in main (argc=<optimized out>, argv=<optimized out>) at
sapi/fpm/fpm/fpm_main.c:1945
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=77289
--
Edit this bug report at https://bugs.php.net/bug.php?id=77289&edit=1