Bug #76339 [Com]: segfault during shutdown

Edit report at https://bugs.php.net/bug.php?id=76339&edit=1 ID: 76339 Comment by: ray dot ward at bigcommerce dot com Reported by: ray dot ward at bigcommerce dot com Summary: segfault during shutdown Status: Open Type: Bug Package: opcache Operating System: Debian 8.10 PHP Version: 7.2.5 Block user comment: N Private report: N New Comment: The zbacktrace provided some insights. We have a shutdown function that is interacting with an object that itself calls gc_collect_cycles. (gdb) zbacktrace [0x7f44baa13260] gc_collect_cycles() [internal function] [0x7f44baa131e0] <redacted>\BatchIndexStrategy->flushUpdateQueue() [0x7f44baa13170] <redacted>\BatchIndexStrategy->flushQueues() [0x7f44baa13100] <redacted>\BatchIndexStrategy->cleanup() [0x7f44baa13030] {closure}() <redacted>:1903 [0x7ffc2eabe640] ??? The object queues updates to be sent to a 3rd party system and flushes them in batches when it reaches a preset size. We also invoke the flush on shutdown to push any remaining/pending updates. We use gc_collect_cycles() as an attempt to force GC and free up memory. Is calling gc_collect_cycles() from php code while php itself is shutting down and GC'ing a bad idea and could lead to these sorts of issues? Previous Comments: ------------------------------------------------------------------------ [2018-05-16 04:32:03] ray dot ward at bigcommerce dot com Also seeing similar segfaults on the PHP 7.0 boxes we haven't upgraded #0 gc_mark_grey (ref=0x0) at ./Zend/zend_gc.c:521 #1 0x00000000005eabd8 in gc_mark_roots () at ./Zend/zend_gc.c:548 #2 zend_gc_collect_cycles () at ./Zend/zend_gc.c:1025 #3 0x00000000005d73bd in zif_gc_collect_cycles (execute_data=<optimized out>, return_value=0x7f44baa13240) at ./Zend/zend_builtin_functions.c:413 #4 0x0000000000646b1d in ZEND_DO_FCALL_BY_NAME_SPEC_HANDLER () at ./Zend/zend_vm_execute.h:714 #5 0x000000000060205b in execute_ex (ex=<optimized out>) at ./Zend/zend_vm_execute.h:414 #6 0x00000000005b21f5 in zend_call_function (fci=fci@entry=0x7ffc2eabe6e0, fci_cache=0x7f444687b340, fci_cache@entry=0x0) at ./Zend/zend_execute_API.c:867 #7 0x00000000005b2629 in call_user_function_ex (function_table=<optimized out>, object=object@entry=0x0, function_name=<optimized out>, retval_ptr=retval_ptr@entry=0x7ffc2eabe770, param_count=<optimized out>, params=<optimized out>, no_separation=1, symbol_table=0x0) at ./Zend/zend_execute_API.c:675 #8 0x00000000005b265d in call_user_function (function_table=<optimized out>, object=object@entry=0x0, function_name=<optimized out>, retval_ptr=retval_ptr@entry=0x7ffc2eabe770, param_count=<optimized out>, params=<optimized out>) at ./Zend/zend_execute_API.c:657 #9 0x00000000004feaf0 in user_shutdown_function_call (zv=<optimized out>) at ./ext/standard/basic_functions.c:4921 #10 0x00000000005d3a0c in zend_hash_apply (ht=0x7f44baa862a0, apply_func=apply_func@entry=0x4fea10 <user_shutdown_function_call>) at ./Zend/zend_hash.c:1537 #11 0x0000000000501f86 in php_call_shutdown_functions () at ./ext/standard/basic_functions.c:5005 #12 0x000000000055f9e5 in php_request_shutdown (dummy=<optimized out>) at ./main/main.c:1796 #13 0x0000000000444cf3 in main (argc=27158462, argv=0x19e6775) at ./sapi/fpm/fpm/fpm_main.c:1995 ------------------------------------------------------------------------ [2018-05-16 00:06:57] ray dot ward at bigcommerce dot com I wasn't able to reproduce the segfault while using valgrind. opcache config: opcache.enable=1 opcache.enable_cli=0 opcache.memory_consumption=1024M opcache.interned_strings_buffer=64 opcache.max_accelerated_files=40000 opcache.revalidate_path=1 opcache.force_restart_timeout=45 opcache.error_log=/var/log/php/opcache.log opcache.log_verbosity_level=2 extensions: apcu apcu_bc bcmath calendar ctype curl dom exif fileinfo ftp gd gettext gmp iconv imagick imap intl json mbstring mysqli mysqlnd newrelic opcache pdo pdo_mysql phar posix pspell readline realpath_turbo shmop simplexml soap sockets sysvmsg sysvsem sysvshm tokenizer wddx xml xmlreader xmlrpc xmlwriter xsl zip We also had the grpc extension installed previously and found it was contributing to a substantial number of segaults and have since removed that. We are still getting some segfaults but with a slightly different trace: #0 gc_mark_grey (ref=0x0) at ./Zend/zend_gc.c:562 #1 0x0000000000606ce8 in gc_mark_roots () at ./Zend/zend_gc.c:583 #2 zend_gc_collect_cycles () at ./Zend/zend_gc.c:1057 #3 0x0000000000606921 in gc_possible_root (ref=0x0) at ./Zend/zend_gc.c:286 #4 0x00000000005eebfd in gc_check_possible_root (ref=<optimized out>) at ./Zend/zend_gc.h:158 #5 i_zval_ptr_dtor (zval_ptr=0x7fe022eec2c0, zval_ptr=0x7fe022eec2c0) at ./Zend/zend_variables.h:51 #6 zend_array_destroy (ht=0x7fe022ee4968) at ./Zend/zend_hash.c:1304 #7 0x000000000061ba8b in zend_objects_store_free_object_storage (objects=0x0, objects@entry=0xa0e0b8 <executor_globals+824>, fast_shutdown=8 '\b', fast_shutdown@entry=1 '\001') at ./Zend/zend_objects_API.c:105 #8 0x00000000005cb034 in shutdown_executor () at ./Zend/zend_execute_API.c:265 #9 0x00000000005dccbb in zend_deactivate () at ./Zend/zend.c:1036 #10 0x0000000000576e0a in php_request_shutdown (dummy=<optimized out>) at ./main/main.c:1904 #11 0x000000000042dd78 in main (argc=37669494, argv=0x23eca33) at ./sapi/fpm/fpm/fpm_main.c:1994 Common theme seems to be ref=0x0 coming from gc_check_possible_root or i_zval_ptr_dtor, both which are inlined functions which makes it harder to debug. ------------------------------------------------------------------------ [2018-05-14 08:58:03] nikic@php.net Can you please try running PHP under valgrind using "USE_ZEND_ALLOC=0 valgrind php script.php" and post the resulting log? If you you can only reproduce under FPM, it should be possible to run FPM using "USE_ZEND_ALLOC=0 valgrind --trace-children=yes php-fpm". Also, can you provide your opcache configuration (e.g. do you use file cache?) and which extensions you have enabled? ------------------------------------------------------------------------ [2018-05-14 07:28:31] ray dot ward at bigcommerce dot com disabling either of these flags for opcache.optimization_level makes the segfault go away ZEND_OPTIMIZER_PASS_1 (1<<0) /* CSE, STRING construction */ ZEND_OPTIMIZER_PASS_11 (1<<10) /* Merge equal constants */ ------------------------------------------------------------------------ [2018-05-14 05:12:30] ray dot ward at bigcommerce dot com Description: ------------ We're experiencing occasional segfaults during php shutdown php-fpm7.2[12494]: segfault at 0 ip 0000000000605e5b sp 00007fff78ece470 error 6 in php-fpm7.2[400000+35e000] Unfortunately I don't have much to go on apart from a GDB trace: Program received signal SIGSEGV, Segmentation fault. gc_mark_grey (ref=0x0) at ./Zend/zend_gc.c:562 #0 gc_mark_grey (ref=0x0) at ./Zend/zend_gc.c:562 #1 0x0000000000606ce8 in gc_mark_roots () at ./Zend/zend_gc.c:583 #2 zend_gc_collect_cycles () at ./Zend/zend_gc.c:1057 #3 0x0000000000606921 in gc_possible_root (ref=0x0) at ./Zend/zend_gc.c:286 #4 0x00000000005cab18 in gc_check_possible_root (ref=<optimized out>) at ./Zend/zend_gc.h:158 #5 i_zval_ptr_dtor (zval_ptr=0x7f318cfb6320, zval_ptr=0x7f318cfb6320) at ./Zend/zend_variables.h:51 #6 _zval_ptr_dtor (zval_ptr=zval_ptr@entry=0x7f318cfb6320) at ./Zend/zend_execute_API.c:532 #7 0x00000000004e633c in spl_object_storage_dtor (element=<optimized out>) at ./ext/spl/spl_observer.c:153 #8 0x00000000005ee825 in zend_hash_destroy (ht=ht@entry=0x7f3211c7c640) at ./Zend/zend_hash.c:1234 #9 0x00000000004e630a in spl_SplObjectStorage_free_storage (object=0x7f3211c7c6a8) at ./ext/spl/spl_observer.c:112 #10 0x000000000061ba8b in zend_objects_store_free_object_storage (objects=0x0, objects@entry=0xa0e0b8 <executor_globals+824>, fast_shutdown=152 '\230', fast_shutdown@entry=1 '\001') at ./Zend/zend_objects_API.c:105 #11 0x00000000005cb034 in shutdown_executor () at ./Zend/zend_execute_API.c:265 #12 0x00000000005dccbb in zend_deactivate () at ./Zend/zend.c:1036 #13 0x0000000000576e0a in php_request_shutdown (dummy=<optimized out>) at ./main/main.c:1904 #14 0x000000000042dd78 in main (argc=17133702, argv=0x1057043) at ./sapi/fpm/fpm/fpm_main.c:1994 _May_ be opcache related, disabling opcache eliminated the segfault. We rely on complex shutdown functions. Perhaps an object being referenced in a function has been GC'd already? ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=76339&edit=1