Edit report at https://bugs.php.net/bug.php?id=77955&edit=1
ID: 77955
Comment by: sjon at hortensius dot net
Reported by: victoria at sharksmedia dot dk
Summary: Random segmentation fault in mysqlnd from php-fpm
Status: Open
Type: Bug
Package: PDO MySQL
Operating System: CentOS 7.6.1810
PHP Version: 7.3.4
Block user comment: N
Private report: N
New Comment:
You could try reverting https://github.com/php/php-src/commit/5eb1f92f31cafc48384f9096012f421b37f6d425
locally (at least in ext/mysqlnd/mysqlnd_result_meta.c) as that seems the most relevant change from
your backtrace and only appeared in 7.3
Previous Comments:
------------------------------------------------------------------------
[2019-05-01 12:36:28] victoria at sharksmedia dot dk
php-fpm error log excerpt:
[01-May-2019 05:50:26] WARNING: [pool bm_live] child 24492 exited on signal 11 (SIGSEGV - core
dumped) after 424.195966 seconds from start
[01-May-2019 05:57:36] WARNING: [pool bm_live] child 24494 exited on signal 11 (SIGSEGV - core
dumped) after 854.271177 seconds from start
[01-May-2019 05:58:29] WARNING: [pool bm_live] child 28154 exited on signal 11 (SIGSEGV - core
dumped) after 53.852203 seconds from start
[01-May-2019 06:03:36] WARNING: [pool bm_live] child 28484 exited on signal 11 (SIGSEGV - core
dumped) after 294.918476 seconds from start
[01-May-2019 06:08:02] WARNING: [pool bm_live] child 31927 exited on signal 11 (SIGSEGV - core
dumped) after 0.909109 seconds from start
Just to state what might be obvious; we have not been able to identify the peice of php code that is
responsible for the segfault, the fault is too rare.
------------------------------------------------------------------------
[2019-05-01 12:04:05] victoria at sharksmedia dot dk
Description:
------------
We are running a medium sized ecommerce site on a custom code base, and since php 7.3.1 was released
in january, we have been trying to upgrade from 7.2.x to newest 7.3.x but have faced random
segmentation faults on each release.
With a backend traffic in the early moning hours of 20-40 req/sec we see ~8-12 segmentation faults
per hour (php-fpm workers). Despite having tried hard we are unable to trigger the fault on demand,
the segmentation fault is almost exclusively triggered by visiting customers seemingly at random.
We are using the remi repository for all php modules.
Maybe relevant:
We are seeing a lot of "Allowed memory size of x bytes exhausted" in database intensive
cronjobs when upgrading, hinting at a memory related issue with pdo/mysqlnd but no segmentation
faults in cronjobs.
Everything runs smoothly on 7.2.18
Relevant Versions
PHP: 7.3.4
PDO MySQL / Mysqlnd: mysqlnd 5.0.12-dev
Installed extensions
php-pecl-scrypt
php-cli
php-gd
php-pecl-igbinary
php-pecl-imagick
php-pecl-vld
php-pecl-uuid
php-pdo
php-xml
php-pecl-hrtime
php-pecl-ssh2
php-json
php-process
php-sodium
php-mbstring
php-brotli
php-pecl-oauth
php-common
php-intl
php-opcache
php-bcmath
php-soap
php-pecl-redis4
php-pecl-varnish
php-mysqlnd
php-pecl-xdebug
php-snuffleupagus
php-fpm
php-gmp
We thought we were affected by following but apparently not, but it sounds similar (we are using a
mix of buffered and unbuffered queries, with native prepared statements)
https://bugs.php.net/bug.php?id=77308
https://bugs.php.net/bug.php?id=77430
Thanks for your time, it is greatly appreciated.
Actual result:
--------------
Bactrace with xDebug disabled:
# gdb /usr/sbin/php-fpm core.31927
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/php-fpm...Reading symbols from
/usr/lib/debug/usr/sbin/php-fpm.debug...done.
done.
[New LWP 31927]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `php-fpm: pool bm_live '.
Program terminated with signal 11, Segmentation fault.
#0 php_mysqlnd_free_field_metadata (meta=0x7f4929401018) at
/usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result_meta.c:38
38 if (meta->sname) {
(gdb) bt
#0 php_mysqlnd_free_field_metadata (meta=0x7f4929401018) at
/usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result_meta.c:38
#1 mysqlnd_mysqlnd_res_meta_free_pub (meta=0x7f492977a718) at
/usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result_meta.c:106
#2 0x00007f49392c3daa in mysqlnd_mysqlnd_res_free_result_contents_internal_pub
(result=0x7f492977a048) at /usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result.c:300
#3 0x00007f49392c46a0 in mysqlnd_mysqlnd_res_free_result_internal_pub (result=0x7f492977a048) at
/usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result.c:316
#4 0x00007f49392c444b in mysqlnd_mysqlnd_res_free_result_pub (result=0x7f492977a048,
implicit=<optimized out>) at /usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result.c:1498
#5 0x00007f49363fc9d9 in pdo_mysql_stmt_cursor_closer (stmt=<optimized out>) at
/usr/src/debug/php-7.3.4/ext/pdo_mysql/mysql_statement.c:901
#6 0x00007f4939098381 in zim_PDOStatement_closeCursor (execute_data=<optimized out>,
return_value=0x7ffc13242710) at /usr/src/debug/php-7.3.4/ext/pdo/pdo_stmt.c:2089
#7 0x000055ced84959c1 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at
/usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:980
#8 execute_ex (ex=0x7f492977a718) at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55485
#9 0x000055ced8497e93 in zend_execute (op_array=op_array@entry=0x7f4948e7c000, return_value=0x0,
return_value@entry=0x7f492a3d84d0) at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:60881
#10 0x000055ced8408bc2 in zend_execute_scripts (type=type@entry=8, retval=0x7f492a3d84d0,
retval@entry=0x0, file_count=1222766048, file_count@entry=3) at
/usr/src/debug/php-7.3.4/Zend/zend.c:1568
#11 0x000055ced83a88c0 in php_execute_script (primary_file=primary_file@entry=0x7ffc13244d20) at
/usr/src/debug/php-7.3.4/main/main.c:2630
#12 0x000055ced820cc0b in main (argc=<optimized out>, argv=<optimized out>) at
/usr/src/debug/php-7.3.4/sapi/fpm/fpm/fpm_main.c:1950
(gdb)
Backtrace with xDebug enabled (our production setup)
gdb /usr/sbin/php-fpm core.24492
GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-redhat-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from /usr/sbin/php-fpm...Reading symbols from
/usr/lib/debug/usr/sbin/php-fpm.debug...done.
done.
[New LWP 24492]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib64/libthread_db.so.1".
Core was generated by `php-fpm: pool bm_live '.
Program terminated with signal 11, Segmentation fault.
#0 php_mysqlnd_free_field_metadata (meta=0x7faee1e01018) at
/usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result_meta.c:38
38 if (meta->sname) {
(gdb) bt
#0 php_mysqlnd_free_field_metadata (meta=0x7faee1e01018) at
/usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result_meta.c:38
#1 mysqlnd_mysqlnd_res_meta_free_pub (meta=0x7faee21fc718) at
/usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result_meta.c:106
#2 0x00007faef327ddaa in mysqlnd_mysqlnd_res_free_result_contents_internal_pub
(result=0x7faee21fc048) at /usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result.c:300
#3 0x00007faef327e6a0 in mysqlnd_mysqlnd_res_free_result_internal_pub (result=0x7faee21fc048) at
/usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result.c:316
#4 0x00007faef327e44b in mysqlnd_mysqlnd_res_free_result_pub (result=0x7faee21fc048,
implicit=<optimized out>) at /usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result.c:1498
#5 0x00007faef03b69d9 in pdo_mysql_stmt_cursor_closer (stmt=<optimized out>) at
/usr/src/debug/php-7.3.4/ext/pdo_mysql/mysql_statement.c:901
#6 0x00007faef3052381 in zim_PDOStatement_closeCursor (execute_data=<optimized out>,
return_value=0x7ffe05bfa1c8) at /usr/src/debug/php-7.3.4/ext/pdo/pdo_stmt.c:2089
#7 0x00007faefc61ce75 in xdebug_execute_internal (current_execute_data=0x7faf0301da00,
return_value=0x7ffe05bfa1c8) at /usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:2050
#8 0x0000557196c3891d in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at
/usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:982
#9 execute_ex (ex=0x7faee21fc718) at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55485
#10 0x00007faf030b16b8 in ?? ()
#11 0x0000557196e321d2 in zend_llist_apply_with_argument (l=<optimized out>,
func=0x557196c3891d <execute_ex+4292297949>, arg=0x7faee43bd9d0) at
/usr/src/debug/php-7.3.4/Zend/zend_llist.c:234
#12 0x00007faefc84fba0 in ?? () from /usr/lib64/php/modules/xdebug.so
#13 0x00005571991df130 in ?? ()
#14 0x0000557196ec6768 in execute_ex (ex=0x7faee21fc718) at
/usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55557
#15 0x00007faefc61c5dd in xdebug_execute_ex (execute_data=0x7faf0301d930) at
/usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:1928
#16 0x0000557196c38be8 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER () at
/usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:1083
#17 0x0000557196ec6768 in execute_ex (ex=0x7faee21fc718) at
/usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55557
#18 0x00007faefc61c5dd in xdebug_execute_ex (execute_data=0x7faf0301d830) at
/usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:1928
#19 0x0000557196c38be8 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER () at
/usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:1083
#20 0x0000557196ec6768 in execute_ex (ex=0x7faee21fc718) at
/usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55557
#21 0x00007faefc61c5dd in xdebug_execute_ex (execute_data=0x7faf0301d780) at
/usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:1928
#22 0x0000557196c38be8 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER () at
/usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:1083
#23 0x0000557196ec6768 in execute_ex (ex=0x7faee21fc718) at
/usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55557
#24 0x00007faefc61c5dd in xdebug_execute_ex (execute_data=0x7faf0301d680) at
/usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:1928
#25 0x0000557196c388aa in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at
/usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:961
#26 0x0000557196ec6768 in execute_ex (ex=0x7faee21fc718) at
/usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55557
#27 0x00007faefc61c5dd in xdebug_execute_ex (execute_data=0x7faf0301d590) at
/usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:1928
#28 0x0000557196c388aa in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at
/usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:961
#29 0x0000557196ec6768 in execute_ex (ex=0x7faee21fc718) at
/usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55557
#30 0x00007faefc61c5dd in xdebug_execute_ex (execute_data=0x7faf0301d490) at
/usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:1928
#31 0x0000557196ebfeac in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER () at
/usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:37708
#32 0x0000557196ec6768 in execute_ex (ex=0x7faee21fc718) at
/usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55557
#33 0x00007faefc61c5dd in xdebug_execute_ex (execute_data=0x7faf0301d1e0) at
/usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:1928
#34 0x0000557196c388aa in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at
/usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:961
#35 0x0000557196ec6768 in execute_ex (ex=0x7faee21fc718) at
/usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55557
#36 0x00007faefc61c5dd in xdebug_execute_ex (execute_data=0x7faf0301d030) at
/usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:1928
#37 0x0000557196ecce93 in zend_execute (op_array=op_array@entry=0x7faf03061380,
return_value=return_value@entry=0x0) at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:60881
#38 0x0000557196e3dbc2 in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0,
file_count=file_count@entry=3) at /usr/src/debug/php-7.3.4/Zend/zend.c:1568
#39 0x0000557196ddd8c0 in php_execute_script (primary_file=primary_file@entry=0x7ffe05bfd220) at
/usr/src/debug/php-7.3.4/main/main.c:2630
#40 0x0000557196c41c0b in main (argc=<optimized out>, argv=<optimized out>) at
/usr/src/debug/php-7.3.4/sapi/fpm/fpm/fpm_main.c:1950
(gdb)
------------------------------------------------------------------------
--
Edit this bug report at https://bugs.php.net/bug.php?id=77955&edit=1