Bug #74403 [Fbk]: Sporadic FPM segfaults

Edit report at https://bugs.php.net/bug.php?id=74403&edit=1 ID: 74403 Updated by: tony2001@php.net Reported by: tony2001@php.net Summary: Sporadic FPM segfaults Status: Feedback Type: Bug Package: PCRE related Operating System: Linux PHP Version: 7.1.4RC1 Assigned To: laruence Block user comment: N Private report: N New Comment: Reverted the attached patch in my private branch and deployed FPM on one production server. After about 6 hours I've got several cores with the backtraces below. Number of segfaults on several hundred servers since March 22: 0. backtrace #1: #0 zend_hash_find_bucket (ht=ht@entry=0xdd28e1cadaf9342c, ht=ht@entry=0xdd28e1cadaf9342c, key=0x7f37b66bdf00, key@entry=0x7f57b453b718) at /home/tony/php-build/php-src/Zend/zend_hash.c:496 #1 zend_hash_find (ht=ht@entry=0x7f37b4539720, key=key@entry=0x7f37b66bdf00) at /home/tony/php-build/php-src/Zend/zend_hash.c:1988 #2 0x00000000006da2e6 in zend_hash_find_ind (key=0x7f37b66bdf00, ht=0x7f37b4539720) at /home/tony/php-build/php-src/Zend/zend_hash.h:278 #3 ZEND_ISSET_ISEMPTY_DIM_OBJ_SPEC_CV_CV_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:37856 #4 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414 #5 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c18720) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481 #6 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800 #7 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414 #8 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c18660) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481 #9 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800 #10 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414 #11 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c18480) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481 #12 0x000000000065b919 in zend_call_function (fci=fci@entry=0x7ffcffa5d8d0, fci_cache=fci_cache@entry=0x7ffcffa5d8a0) at /home/tony/php-build/php-src/Zend/zend_execute_API.c:858 #13 0x0000000000689db3 in zend_call_method (object=0x0, obj_ce=<optimized out>, fn_proxy=0x7f3833c55870, function_name=0x7f3833c74248 "main_autoload", function_name_len=<optimized out>, retval_ptr=retval_ptr@entry=0x0, param_count=1, arg1=0x7f3833c18470, arg2=0x0) at /home/tony/php-build/php-src/Zend/zend_interfaces.c:104 #14 0x0000000000525c4e in zif_spl_autoload_call (execute_data=<optimized out>, return_value=<optimized out>) at /home/tony/php-build/php-src/ext/spl/php_spl.c:421 #15 0x00007f382e4ca680 in memtrack_execute_internal (current_execute_data=0x7f3833c18410, return_value=0x7ffcffa5dbc0) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:563 #16 0x000000000065bb57 in zend_call_function (fci=fci@entry=0x7ffcffa5dc00, fci_cache=fci_cache@entry=0x7ffcffa5dbd0) at /home/tony/php-build/php-src/Zend/zend_execute_API.c:880 #17 0x000000000065bee2 in zend_lookup_class_ex (name=name@entry=0x7f37b66bdf00, key=key@entry=0x0, use_autoload=use_autoload@entry=1) at /home/tony/php-build/php-src/Zend/zend_execute_API.c:1040 #18 0x000000000065c635 in zend_fetch_class (class_name=0x7f37b66bdf00, fetch_type=512) at /home/tony/php-build/php-src/Zend/zend_execute_API.c:1364 #19 0x00000000006e3d2a in ZEND_FETCH_CLASS_SPEC_CV_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:2330 #20 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414 #21 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c18070) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481 #22 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800 #23 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414 #24 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c14e60) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481 #25 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800 #26 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414 #27 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c134c0) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481 #28 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800 (gdb) p p $1 = (Bucket *) 0x7f57b453b718 (gdb) p *p Cannot access memory at address 0x7f57b453b718 backtrace #2: #0 zend_verify_abstract_class (ce=0x7f37a95d414000) at /home/tony/php-build/php-src/Zend/zend_execute_API.c:1441 #1 0x00000000006b6f24 in ZEND_VERIFY_ABSTRACT_CLASS_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:1539 #2 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414 #3 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c15af0) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481 #4 0x000000000071673c in ZEND_INCLUDE_OR_EVAL_SPEC_CV_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:29532 #5 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414 #6 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c15720) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481 #7 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800 #8 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414 #9 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c156a0) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481 #10 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800 #11 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414 #12 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c154c0) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481 #13 0x000000000065b919 in zend_call_function (fci=fci@entry=0x7ffcffa5d0c0, fci_cache=fci_cache@entry=0x7ffcffa5d090) at /home/tony/php-build/php-src/Zend/zend_execute_API.c:858 #14 0x0000000000689db3 in zend_call_method (object=0x0, obj_ce=<optimized out>, fn_proxy=0x7f3833c55810, function_name=0x7f3833c762e8 "main_autoload", function_name_len=<optimized out>, retval_ptr=retval_ptr@entry=0x0, param_count=1, arg1=0x7f3833c154b0, arg2=0x0) at /home/tony/php-build/php-src/Zend/zend_interfaces.c:104 #15 0x0000000000525c4e in zif_spl_autoload_call (execute_data=<optimized out>, return_value=<optimized out>) at /home/tony/php-build/php-src/ext/spl/php_spl.c:421 #16 0x00007f382e4ca680 in memtrack_execute_internal (current_execute_data=0x7f3833c15450, return_value=0x7ffcffa5d3b0) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:563 #17 0x000000000065bb57 in zend_call_function (fci=fci@entry=0x7ffcffa5d3f0, fci_cache=fci_cache@entry=0x7ffcffa5d3c0) at /home/tony/php-build/php-src/Zend/zend_execute_API.c:880 #18 0x000000000065bee2 in zend_lookup_class_ex (name=name@entry=0x7f37b4487cd0, key=0x7f37b4487b98, use_autoload=use_autoload@entry=1) at /home/tony/php-build/php-src/Zend/zend_execute_API.c:1040 #19 0x000000000065c7d8 in zend_fetch_class_by_name (class_name=0x7f37b4487cd0, key=<optimized out>, fetch_type=fetch_type@entry=512) at /home/tony/php-build/php-src/Zend/zend_execute_API.c:1386 #20 0x00000000007148d7 in ZEND_INIT_STATIC_METHOD_CALL_SPEC_CONST_CONST_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:5667 #21 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414 #22 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c152e0) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481 #23 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800 #24 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414 #25 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c15180) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481 #26 0x00000000006f115d in ZEND_DO_FCALL_SPEC_HANDLER () at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:800 #27 0x00000000006b0e4b in execute_ex (ex=<optimized out>) at /home/tony/php-build/php-src/Zend/zend_vm_execute.h:414 #28 0x00007f382e4ca31f in memtrack_execute_ex (execute_data=0x7f3833c14c30) at /home/tony/php-build/php-src/ext/memtrack/memtrack.c:481 (gdb) p *ce Cannot access memory at address 0x7f37a95d414000 (gdb) p ce $1 = (zend_class_entry *) 0x7f37a95d414000 Previous Comments: ------------------------------------------------------------------------ [2017-04-10 11:45:23] laruence@php.net Thank you for this bug report. To properly diagnose the problem, we need a backtrace to see what is happening behind the scenes. To find out how to generate a backtrace, please read http://bugs.php.net/bugs-generating-backtrace.php for *NIX and http://bugs.php.net/bugs-generating-backtrace-win32.php for Win32 Once you have generated a backtrace, please submit it to this bug report and change the status back to "Open". Thank you for helping us make PHP better. as we discussed early in mails, I don't think that diretctly relates to that change, anyway, could you please paste the backtrace out? thanks ------------------------------------------------------------------------ [2017-04-10 11:22:45] tony2001@php.net https://bugs.php.net/bug.php?id=73740 - this one looks familiar. ------------------------------------------------------------------------ [2017-04-10 11:17:26] requinix@php.net There have been a few php-fpm crashes reported over the last couple months. Can you tell if any of them are related to this? ------------------------------------------------------------------------ [2017-04-10 11:09:09] tony2001@php.net The following patch has been added/updated: Patch Name: pcre-crashes-fix.diff Revision: 1491822546 URL: https://bugs.php.net/patch-display.php?bug=74403&patch=pcre-crashes-fix.diff&revision=1491822546 ------------------------------------------------------------------------ [2017-04-10 11:08:35] tony2001@php.net Description: ------------ Patch ebfd93f725eb9b9cfadbbde98047efe76f658da6 introduced an issue that is both hard to debug and reproduce. FPM processes start to crash in completely random places (mostly pointing to the executor code), all FPM children at once. Unfortunately, I don't have a reproduce case and I don't know of any reliable way to reproduce it, other than to start FPM on production server under a heavy load and leave it for a while. I've spent a lot of time debugging it and have found that the problem goes away as soon as I revert ebfd93f725eb9b9cfadbbde98047efe76f658da6. After looking at the patch closely, I've noticed that the patch changes one tricky part of the code managing refcount of the string stored in the cache. In the new version that part is gone and reverting this part of the patch fixes the issue, see attached patch. Test script: --------------- . Expected result: ---------------- No crashes. Actual result: -------------- Sporadic FPM segfaults. ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=74403&edit=1