Bug #77955 [NEW]: Random segmentation fault in mysqlnd from php-fpm

From: victoria at sharksmedia dot dk Operating system: CentOS 7.6.1810 PHP version: 7.3.4 Package: PDO MySQL Bug Type: Bug Bug description:Random segmentation fault in mysqlnd from php-fpm Description: ------------ We are running a medium sized ecommerce site on a custom code base, and since php 7.3.1 was released in january, we have been trying to upgrade from 7.2.x to newest 7.3.x but have faced random segmentation faults on each release. With a backend traffic in the early moning hours of 20-40 req/sec we see ~8-12 segmentation faults per hour (php-fpm workers). Despite having tried hard we are unable to trigger the fault on demand, the segmentation fault is almost exclusively triggered by visiting customers seemingly at random. We are using the remi repository for all php modules. Maybe relevant: We are seeing a lot of "Allowed memory size of x bytes exhausted" in database intensive cronjobs when upgrading, hinting at a memory related issue with pdo/mysqlnd but no segmentation faults in cronjobs. Everything runs smoothly on 7.2.18 Relevant Versions PHP: 7.3.4 PDO MySQL / Mysqlnd: mysqlnd 5.0.12-dev Installed extensions php-pecl-scrypt php-cli php-gd php-pecl-igbinary php-pecl-imagick php-pecl-vld php-pecl-uuid php-pdo php-xml php-pecl-hrtime php-pecl-ssh2 php-json php-process php-sodium php-mbstring php-brotli php-pecl-oauth php-common php-intl php-opcache php-bcmath php-soap php-pecl-redis4 php-pecl-varnish php-mysqlnd php-pecl-xdebug php-snuffleupagus php-fpm php-gmp We thought we were affected by following but apparently not, but it sounds similar (we are using a mix of buffered and unbuffered queries, with native prepared statements) https://bugs.php.net/bug.php?id=77308 https://bugs.php.net/bug.php?id=77430 Thanks for your time, it is greatly appreciated. Actual result: -------------- Bactrace with xDebug disabled: # gdb /usr/sbin/php-fpm core.31927 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/php-fpm...Reading symbols from /usr/lib/debug/usr/sbin/php-fpm.debug...done. done. [New LWP 31927] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `php-fpm: pool bm_live '. Program terminated with signal 11, Segmentation fault. #0 php_mysqlnd_free_field_metadata (meta=0x7f4929401018) at /usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result_meta.c:38 38 if (meta->sname) { (gdb) bt #0 php_mysqlnd_free_field_metadata (meta=0x7f4929401018) at /usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result_meta.c:38 #1 mysqlnd_mysqlnd_res_meta_free_pub (meta=0x7f492977a718) at /usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result_meta.c:106 #2 0x00007f49392c3daa in mysqlnd_mysqlnd_res_free_result_contents_internal_pub (result=0x7f492977a048) at /usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result.c:300 #3 0x00007f49392c46a0 in mysqlnd_mysqlnd_res_free_result_internal_pub (result=0x7f492977a048) at /usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result.c:316 #4 0x00007f49392c444b in mysqlnd_mysqlnd_res_free_result_pub (result=0x7f492977a048, implicit=<optimized out>) at /usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result.c:1498 #5 0x00007f49363fc9d9 in pdo_mysql_stmt_cursor_closer (stmt=<optimized out>) at /usr/src/debug/php-7.3.4/ext/pdo_mysql/mysql_statement.c:901 #6 0x00007f4939098381 in zim_PDOStatement_closeCursor (execute_data=<optimized out>, return_value=0x7ffc13242710) at /usr/src/debug/php-7.3.4/ext/pdo/pdo_stmt.c:2089 #7 0x000055ced84959c1 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:980 #8 execute_ex (ex=0x7f492977a718) at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55485 #9 0x000055ced8497e93 in zend_execute (op_array=op_array@entry=0x7f4948e7c000, return_value=0x0, return_value@entry=0x7f492a3d84d0) at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:60881 #10 0x000055ced8408bc2 in zend_execute_scripts (type=type@entry=8, retval=0x7f492a3d84d0, retval@entry=0x0, file_count=1222766048, file_count@entry=3) at /usr/src/debug/php-7.3.4/Zend/zend.c:1568 #11 0x000055ced83a88c0 in php_execute_script (primary_file=primary_file@entry=0x7ffc13244d20) at /usr/src/debug/php-7.3.4/main/main.c:2630 #12 0x000055ced820cc0b in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/php-7.3.4/sapi/fpm/fpm/fpm_main.c:1950 (gdb) Backtrace with xDebug enabled (our production setup) gdb /usr/sbin/php-fpm core.24492 GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-114.el7 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>... Reading symbols from /usr/sbin/php-fpm...Reading symbols from /usr/lib/debug/usr/sbin/php-fpm.debug...done. done. [New LWP 24492] [Thread debugging using libthread_db enabled] Using host libthread_db library "/lib64/libthread_db.so.1". Core was generated by `php-fpm: pool bm_live '. Program terminated with signal 11, Segmentation fault. #0 php_mysqlnd_free_field_metadata (meta=0x7faee1e01018) at /usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result_meta.c:38 38 if (meta->sname) { (gdb) bt #0 php_mysqlnd_free_field_metadata (meta=0x7faee1e01018) at /usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result_meta.c:38 #1 mysqlnd_mysqlnd_res_meta_free_pub (meta=0x7faee21fc718) at /usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result_meta.c:106 #2 0x00007faef327ddaa in mysqlnd_mysqlnd_res_free_result_contents_internal_pub (result=0x7faee21fc048) at /usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result.c:300 #3 0x00007faef327e6a0 in mysqlnd_mysqlnd_res_free_result_internal_pub (result=0x7faee21fc048) at /usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result.c:316 #4 0x00007faef327e44b in mysqlnd_mysqlnd_res_free_result_pub (result=0x7faee21fc048, implicit=<optimized out>) at /usr/src/debug/php-7.3.4/ext/mysqlnd/mysqlnd_result.c:1498 #5 0x00007faef03b69d9 in pdo_mysql_stmt_cursor_closer (stmt=<optimized out>) at /usr/src/debug/php-7.3.4/ext/pdo_mysql/mysql_statement.c:901 #6 0x00007faef3052381 in zim_PDOStatement_closeCursor (execute_data=<optimized out>, return_value=0x7ffe05bfa1c8) at /usr/src/debug/php-7.3.4/ext/pdo/pdo_stmt.c:2089 #7 0x00007faefc61ce75 in xdebug_execute_internal (current_execute_data=0x7faf0301da00, return_value=0x7ffe05bfa1c8) at /usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:2050 #8 0x0000557196c3891d in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:982 #9 execute_ex (ex=0x7faee21fc718) at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55485 #10 0x00007faf030b16b8 in ?? () #11 0x0000557196e321d2 in zend_llist_apply_with_argument (l=<optimized out>, func=0x557196c3891d <execute_ex+4292297949>, arg=0x7faee43bd9d0) at /usr/src/debug/php-7.3.4/Zend/zend_llist.c:234 #12 0x00007faefc84fba0 in ?? () from /usr/lib64/php/modules/xdebug.so #13 0x00005571991df130 in ?? () #14 0x0000557196ec6768 in execute_ex (ex=0x7faee21fc718) at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55557 #15 0x00007faefc61c5dd in xdebug_execute_ex (execute_data=0x7faf0301d930) at /usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:1928 #16 0x0000557196c38be8 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER () at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:1083 #17 0x0000557196ec6768 in execute_ex (ex=0x7faee21fc718) at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55557 #18 0x00007faefc61c5dd in xdebug_execute_ex (execute_data=0x7faf0301d830) at /usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:1928 #19 0x0000557196c38be8 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER () at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:1083 #20 0x0000557196ec6768 in execute_ex (ex=0x7faee21fc718) at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55557 #21 0x00007faefc61c5dd in xdebug_execute_ex (execute_data=0x7faf0301d780) at /usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:1928 #22 0x0000557196c38be8 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER () at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:1083 #23 0x0000557196ec6768 in execute_ex (ex=0x7faee21fc718) at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55557 #24 0x00007faefc61c5dd in xdebug_execute_ex (execute_data=0x7faf0301d680) at /usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:1928 #25 0x0000557196c388aa in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:961 #26 0x0000557196ec6768 in execute_ex (ex=0x7faee21fc718) at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55557 #27 0x00007faefc61c5dd in xdebug_execute_ex (execute_data=0x7faf0301d590) at /usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:1928 #28 0x0000557196c388aa in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:961 #29 0x0000557196ec6768 in execute_ex (ex=0x7faee21fc718) at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55557 #30 0x00007faefc61c5dd in xdebug_execute_ex (execute_data=0x7faf0301d490) at /usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:1928 #31 0x0000557196ebfeac in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER () at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:37708 #32 0x0000557196ec6768 in execute_ex (ex=0x7faee21fc718) at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55557 #33 0x00007faefc61c5dd in xdebug_execute_ex (execute_data=0x7faf0301d1e0) at /usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:1928 #34 0x0000557196c388aa in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:961 #35 0x0000557196ec6768 in execute_ex (ex=0x7faee21fc718) at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:55557 #36 0x00007faefc61c5dd in xdebug_execute_ex (execute_data=0x7faf0301d030) at /usr/src/debug/php-pecl-xdebug-2.7.1/NTS/xdebug.c:1928 #37 0x0000557196ecce93 in zend_execute (op_array=op_array@entry=0x7faf03061380, return_value=return_value@entry=0x0) at /usr/src/debug/php-7.3.4/Zend/zend_vm_execute.h:60881 #38 0x0000557196e3dbc2 in zend_execute_scripts (type=type@entry=8, retval=retval@entry=0x0, file_count=file_count@entry=3) at /usr/src/debug/php-7.3.4/Zend/zend.c:1568 #39 0x0000557196ddd8c0 in php_execute_script (primary_file=primary_file@entry=0x7ffe05bfd220) at /usr/src/debug/php-7.3.4/main/main.c:2630 #40 0x0000557196c41c0b in main (argc=<optimized out>, argv=<optimized out>) at /usr/src/debug/php-7.3.4/sapi/fpm/fpm/fpm_main.c:1950 (gdb) -- Edit bug report at https://bugs.php.net/bug.php?id=77955&edit=1 -- Try a snapshot (PHP 5.4): https://bugs.php.net/fix.php?id=77955&r=trysnapshot54 Try a snapshot (PHP 5.5): https://bugs.php.net/fix.php?id=77955&r=trysnapshot55 Try a snapshot (trunk): https://bugs.php.net/fix.php?id=77955&r=trysnapshottrunk Fixed in SVN: https://bugs.php.net/fix.php?id=77955&r=fixed Fixed in release: https://bugs.php.net/fix.php?id=77955&r=alreadyfixed Need backtrace: https://bugs.php.net/fix.php?id=77955&r=needtrace Need Reproduce Script: https://bugs.php.net/fix.php?id=77955&r=needscript Try newer version: https://bugs.php.net/fix.php?id=77955&r=oldversion Not developer issue: https://bugs.php.net/fix.php?id=77955&r=support Expected behavior: https://bugs.php.net/fix.php?id=77955&r=notwrong Not enough info: https://bugs.php.net/fix.php?id=77955&r=notenoughinfo Submitted twice: https://bugs.php.net/fix.php?id=77955&r=submittedtwice register_globals: https://bugs.php.net/fix.php?id=77955&r=globals PHP 4 support discontinued: https://bugs.php.net/fix.php?id=77955&r=php4 Daylight Savings: https://bugs.php.net/fix.php?id=77955&r=dst IIS Stability: https://bugs.php.net/fix.php?id=77955&r=isapi Install GNU Sed: https://bugs.php.net/fix.php?id=77955&r=gnused Floating point limitations: https://bugs.php.net/fix.php?id=77955&r=float No Zend Extensions: https://bugs.php.net/fix.php?id=77955&r=nozend MySQL Configuration Error: https://bugs.php.net/fix.php?id=77955&r=mysqlcfg