Bug #77443 [Opn->Csd]: FPM hangs on reload due to children handling SIGTERM since PHP 7.1

Edit report at https://bugs.php.net/bug.php?id=77443&edit=1 ID: 77443 Updated by: bukka@php.net Reported by: emayoral at arsys dot es Summary: FPM hangs on reload due to children handling SIGTERM since PHP 7.1 -Status: Open +Status: Closed Type: Bug Package: FPM related Operating System: CentOS 6 PHP Version: 7.2.14 -Assigned To: +Assigned To: bukka Block user comment: N Private report: N New Comment: This issue should be fixed in 7.4. Previous Comments: ------------------------------------------------------------------------ [2019-11-20 04:54:58] mnikulin at plesk dot com Issue with signal blocked by opcache has been likely solved in PHP-7.2.22, see https://github.com/php/php-src/pull/4461 https://github.com/php/php-src/commit/38f1288b6427fc9e2fa2b5ad9912745ded923ee7 and Bug #76895 SIGKILL should be fixed in 7.4 https://github.com/php/php-src/commit/0ed6c3714087b254c49185568df96a31df76b2ce ------------------------------------------------------------------------ [2019-01-24 03:18:18] mnikulin at plesk dot com Related To: Bug #77140 ------------------------------------------------------------------------ [2019-01-22 09:55:07] mnikulin at plesk dot com Please, have a look at Bug #76601 "Partially working php-fpm after incomplete reload" that happens for both 5.6 and 7.x versions and at Bug #76895 "PHP-FPM7.x can't finish reloading after encountering a fatal error" that is specific to 7.x versions and easily reproducible The following one-line patch fix SIGKILL so the problem becomes invisible, however the proper way would be to avoid ignoring of SIGQUIT and SIGTERM by child processes https://bugs.php.net/patch-display.php?bug_id=76601&patch=php-76601_kill-not-rescheduled_7.2.11_2018-09-28.patch&revision=latest ------------------------------------------------------------------------ [2019-01-11 10:26:22] emayoral at arsys dot es Digging further into the code, I think this case is covered in: sapi/fpm/fpm/fpm_process_ctl.c static void fpm_pctl_action_next() /* {{{ */ { int sig, timeout; if (!fpm_globals.running_children) { fpm_pctl_action_last(); } if (fpm_signal_sent == 0) { if (fpm_state == FPM_PCTL_STATE_TERMINATING) { sig = SIGTERM; } else { sig = SIGQUIT; } timeout = fpm_global_config.process_control_timeout; } else { if (fpm_signal_sent == SIGQUIT) { sig = SIGTERM; } else { sig = SIGKILL; } timeout = 1; } fpm_pctl_kill_all(sig); fpm_signal_sent = sig; fpm_pctl_timeout_set(timeout); } (Send SIGQUIT, wait process_control_timeout , send SIGTERM, wait 1 second, send SIGKILL) But my experience is that after the SIGQUIT and SIGTERM, the SIGKILL never happens (Which didn't matter until php 7.1 as SIGTERM would do the job). Not sure why, though. ------------------------------------------------------------------------ [2019-01-11 07:47:05] emayoral at arsys dot es Description: ------------ I run a system with a relatively large number of pfp-fpm pools. Whenever a pool is added or removed a php-fpm reload is issued. The behaviour of php-fpm on reload is to terminate children with SIGQUIT, wait process_control_timeout (10 seconds in my config), then issue SIGTERM to the remaining children. Now, some of my pool children will not die on SIGQUIT when opcache is used, but that is a different bug. Point is until PHP 7.0, SIGTERM would finish the remaining children, the reload would complete in 10 seconds and everything would be fine. But starting with PHP 7.1 the php processes also handle SIGTERM (I suspect it uses the same signal handler as for SIGQUIT, although I am not 100% sure of that). Now the children which did not die on SIGQUIT do not die on SIGTERM either and the reload will not complete until those children are manually issued a SIGKILL. I think php-fpm 7.1 and higher should do its first round of SIGQUIT to its children on reload, but after process_control_timeout, issue a SIGKILL instead of a SIGTERM, since SIGTERM is not an effective way of ending rogue children any more. ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=77443&edit=1