Bug #76339 [Opn]: segfault during shutdown

Edit report at https://bugs.php.net/bug.php?id=76339&edit=1 ID: 76339 Updated by: nikic@php.net Reported by: ray dot ward at bigcommerce dot com Summary: segfault during shutdown Status: Open Type: Bug Package: opcache Operating System: Debian 8.10 PHP Version: 7.2.5 Block user comment: N Private report: N New Comment: Can you tell me which version of APCu you are using? The issues you are experiencing are most likely caused by some extension performing incorrect refcounting and it manifesting during GC and shutdown. Unfortunately it doesn't really tell us much about the root cause, which is why a valgrind trace would be useful. When you say it does not reproduce under valgrind, does this refer only to the segfault, or is the valgrind output completely clean (no "invalid read" or similar warnings)? Previous Comments: ------------------------------------------------------------------------ [2018-05-16 05:00:42] ray dot ward at bigcommerce dot com The zbacktrace provided some insights. We have a shutdown function that is interacting with an object that itself calls gc_collect_cycles. (gdb) zbacktrace [0x7f44baa13260] gc_collect_cycles() [internal function] [0x7f44baa131e0] <redacted>\BatchIndexStrategy->flushUpdateQueue() [0x7f44baa13170] <redacted>\BatchIndexStrategy->flushQueues() [0x7f44baa13100] <redacted>\BatchIndexStrategy->cleanup() [0x7f44baa13030] {closure}() <redacted>:1903 [0x7ffc2eabe640] ??? The object queues updates to be sent to a 3rd party system and flushes them in batches when it reaches a preset size. We also invoke the flush on shutdown to push any remaining/pending updates. We use gc_collect_cycles() as an attempt to force GC and free up memory. Is calling gc_collect_cycles() from php code while php itself is shutting down and GC'ing a bad idea and could lead to these sorts of issues? ------------------------------------------------------------------------ [2018-05-16 04:32:03] ray dot ward at bigcommerce dot com Also seeing similar segfaults on the PHP 7.0 boxes we haven't upgraded #0 gc_mark_grey (ref=0x0) at ./Zend/zend_gc.c:521 #1 0x00000000005eabd8 in gc_mark_roots () at ./Zend/zend_gc.c:548 #2 zend_gc_collect_cycles () at ./Zend/zend_gc.c:1025 #3 0x00000000005d73bd in zif_gc_collect_cycles (execute_data=<optimized out>, return_value=0x7f44baa13240) at ./Zend/zend_builtin_functions.c:413 #4 0x0000000000646b1d in ZEND_DO_FCALL_BY_NAME_SPEC_HANDLER () at ./Zend/zend_vm_execute.h:714 #5 0x000000000060205b in execute_ex (ex=<optimized out>) at ./Zend/zend_vm_execute.h:414 #6 0x00000000005b21f5 in zend_call_function (fci=fci@entry=0x7ffc2eabe6e0, fci_cache=0x7f444687b340, fci_cache@entry=0x0) at ./Zend/zend_execute_API.c:867 #7 0x00000000005b2629 in call_user_function_ex (function_table=<optimized out>, object=object@entry=0x0, function_name=<optimized out>, retval_ptr=retval_ptr@entry=0x7ffc2eabe770, param_count=<optimized out>, params=<optimized out>, no_separation=1, symbol_table=0x0) at ./Zend/zend_execute_API.c:675 #8 0x00000000005b265d in call_user_function (function_table=<optimized out>, object=object@entry=0x0, function_name=<optimized out>, retval_ptr=retval_ptr@entry=0x7ffc2eabe770, param_count=<optimized out>, params=<optimized out>) at ./Zend/zend_execute_API.c:657 #9 0x00000000004feaf0 in user_shutdown_function_call (zv=<optimized out>) at ./ext/standard/basic_functions.c:4921 #10 0x00000000005d3a0c in zend_hash_apply (ht=0x7f44baa862a0, apply_func=apply_func@entry=0x4fea10 <user_shutdown_function_call>) at ./Zend/zend_hash.c:1537 #11 0x0000000000501f86 in php_call_shutdown_functions () at ./ext/standard/basic_functions.c:5005 #12 0x000000000055f9e5 in php_request_shutdown (dummy=<optimized out>) at ./main/main.c:1796 #13 0x0000000000444cf3 in main (argc=27158462, argv=0x19e6775) at ./sapi/fpm/fpm/fpm_main.c:1995 ------------------------------------------------------------------------ [2018-05-16 00:06:57] ray dot ward at bigcommerce dot com I wasn't able to reproduce the segfault while using valgrind. opcache config: opcache.enable=1 opcache.enable_cli=0 opcache.memory_consumption=1024M opcache.interned_strings_buffer=64 opcache.max_accelerated_files=40000 opcache.revalidate_path=1 opcache.force_restart_timeout=45 opcache.error_log=/var/log/php/opcache.log opcache.log_verbosity_level=2 extensions: apcu apcu_bc bcmath calendar ctype curl dom exif fileinfo ftp gd gettext gmp iconv imagick imap intl json mbstring mysqli mysqlnd newrelic opcache pdo pdo_mysql phar posix pspell readline realpath_turbo shmop simplexml soap sockets sysvmsg sysvsem sysvshm tokenizer wddx xml xmlreader xmlrpc xmlwriter xsl zip We also had the grpc extension installed previously and found it was contributing to a substantial number of segaults and have since removed that. We are still getting some segfaults but with a slightly different trace: #0 gc_mark_grey (ref=0x0) at ./Zend/zend_gc.c:562 #1 0x0000000000606ce8 in gc_mark_roots () at ./Zend/zend_gc.c:583 #2 zend_gc_collect_cycles () at ./Zend/zend_gc.c:1057 #3 0x0000000000606921 in gc_possible_root (ref=0x0) at ./Zend/zend_gc.c:286 #4 0x00000000005eebfd in gc_check_possible_root (ref=<optimized out>) at ./Zend/zend_gc.h:158 #5 i_zval_ptr_dtor (zval_ptr=0x7fe022eec2c0, zval_ptr=0x7fe022eec2c0) at ./Zend/zend_variables.h:51 #6 zend_array_destroy (ht=0x7fe022ee4968) at ./Zend/zend_hash.c:1304 #7 0x000000000061ba8b in zend_objects_store_free_object_storage (objects=0x0, objects@entry=0xa0e0b8 <executor_globals+824>, fast_shutdown=8 '\b', fast_shutdown@entry=1 '\001') at ./Zend/zend_objects_API.c:105 #8 0x00000000005cb034 in shutdown_executor () at ./Zend/zend_execute_API.c:265 #9 0x00000000005dccbb in zend_deactivate () at ./Zend/zend.c:1036 #10 0x0000000000576e0a in php_request_shutdown (dummy=<optimized out>) at ./main/main.c:1904 #11 0x000000000042dd78 in main (argc=37669494, argv=0x23eca33) at ./sapi/fpm/fpm/fpm_main.c:1994 Common theme seems to be ref=0x0 coming from gc_check_possible_root or i_zval_ptr_dtor, both which are inlined functions which makes it harder to debug. ------------------------------------------------------------------------ [2018-05-14 08:58:03] nikic@php.net Can you please try running PHP under valgrind using "USE_ZEND_ALLOC=0 valgrind php script.php" and post the resulting log? If you you can only reproduce under FPM, it should be possible to run FPM using "USE_ZEND_ALLOC=0 valgrind --trace-children=yes php-fpm". Also, can you provide your opcache configuration (e.g. do you use file cache?) and which extensions you have enabled? ------------------------------------------------------------------------ [2018-05-14 07:28:31] ray dot ward at bigcommerce dot com disabling either of these flags for opcache.optimization_level makes the segfault go away ZEND_OPTIMIZER_PASS_1 (1<<0) /* CSE, STRING construction */ ZEND_OPTIMIZER_PASS_11 (1<<10) /* Merge equal constants */ ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=76339 -- Edit this bug report at https://bugs.php.net/bug.php?id=76339&edit=1