Bug #77023 [Asn]: PHP-FPM cannot shutdown processes

Edit report at https://bugs.php.net/bug.php?id=77023&edit=1 ID: 77023 User updated by: azurit at pobox dot sk Reported by: azurit at pobox dot sk Summary: PHP-FPM cannot shutdown processes Status: Assigned Type: Bug Package: FPM related Operating System: Debian 9 PHP Version: 7.2.11 Assigned To: bukka Block user comment: N Private report: N New Comment: Thanks, i'm really happy someone is going to look into this. Meanwhile, i bypassed the problem using this: - i'm periodically running a script which is looping through the all pools status pages, searching for processes which are idling more than idle timeout and killing them - patched FPM and disabled process respawn if they are killed (by the way, this would be a great official feature) Previous Comments: ------------------------------------------------------------------------ [2022-02-13 20:52:53] bukka@php.net Apology for very late replay. I looked a bit to the code and it seems to me that this might happen if FPM is failing to kill the process using the SIGQUIT that is used for that purpose. In other cases this is actually handled by trying SIGKILL if SIGQUIT wasn't successful but not for the dynamic idle check which seems like an omission to me. Will try to put together a patch in the coming months and see if I can also find some way to recreate this. ------------------------------------------------------------------------ [2018-10-16 11:30:54] azurit at pobox dot sk Description: ------------ I'm not albe to reproduce this problem but it is sometimes happening to pool of one of our customers. Pool si set like this: pm = dynamic pm.max_children = 30 pm.start_servers = 1 pm.min_spare_servers = 1 pm.max_spare_servers = 1 I want from it to keep only one IDLE process at a time but FPM sometimes fails to kill IDLE processes (kill because of pm.max_spare_servers) which results into having 30 IDLE processes (or other number, depends on pm.max_children). When i start killing FPM processes by hand, FPM is starting them back BUT there are always few stucked FPM processes (probably the ones which were running for longest time) which, if are killed, FPM itself starts killing also other IDLE processes (like it was waiting for them to shutdown before start shutdowning other processes) and everything goes to normal. I was able to get a strace output from one such process: socket(AF_UNIX, SOCK_STREAM, 0) = 0 rt_sigreturn({mask=[]}) = 43 accept(26, 0x7ffee4d3f860, [112]) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} --- close(0) = 0 socket(AF_UNIX, SOCK_STREAM, 0) = 0 rt_sigreturn({mask=[]}) = 43 accept(26, 0x7ffee4d3f860, [112]) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} --- close(0) = 0 socket(AF_UNIX, SOCK_STREAM, 0) = 0 rt_sigreturn({mask=[]}) = 43 accept(26, 0x7ffee4d3f860, [112]) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} --- close(0) = 0 socket(AF_UNIX, SOCK_STREAM, 0) = 0 rt_sigreturn({mask=[]}) = 43 accept(26, 0x7ffee4d3f860, [112]) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} --- close(0) = 0 socket(AF_UNIX, SOCK_STREAM, 0) = 0 rt_sigreturn({mask=[]}) = 43 accept(26, 0x7ffee4d3f860, [112]) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} --- close(0) = 0 socket(AF_UNIX, SOCK_STREAM, 0) = 0 rt_sigreturn({mask=[]}) = 43 accept(26, 0x7ffee4d3f860, [112]) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} --- close(0) = 0 socket(AF_UNIX, SOCK_STREAM, 0) = 0 rt_sigreturn({mask=[]}) = 43 accept(26, 0x7ffee4d3f860, [112]) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} --- close(0) = 0 socket(AF_UNIX, SOCK_STREAM, 0) = 0 rt_sigreturn({mask=[]}) = 43 accept(26, 0x7ffee4d3f860, [112]) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} --- close(0) = 0 socket(AF_UNIX, SOCK_STREAM, 0) = 0 rt_sigreturn({mask=[]}) = 43 accept(26, 0x7ffee4d3f860, [112]) = ? ERESTARTSYS (To be restarted if SA_RESTART is set) --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=30660, si_uid=0} --- close(0) = 0 ... Look on this: # ps aux | grep customer1 | grep fpm | wc -l 11 # ps aux | grep customer1 | grep fpm customer1 2345 0.0 0.1 398280 20640 ? S 12:34 0:00 php-fpm: pool customer1 customer1 3295 0.4 0.2 493180 47964 ? S 12:34 0:12 php-fpm: pool customer1 customer1 10887 0.4 0.2 489452 43628 ? S 12:35 0:12 php-fpm: pool customer1 customer1 12018 0.6 0.3 497272 50852 ? S 12:35 0:18 php-fpm: pool customer1 customer1 12707 0.4 0.3 496552 51748 ? S 12:35 0:14 php-fpm: pool customer1 customer1 13157 0.5 0.4 511312 65676 ? S 12:35 0:15 php-fpm: pool customer1 customer1 15860 0.6 0.3 797176 53464 ? S 12:36 0:17 php-fpm: pool customer1 customer1 15941 0.5 0.3 788804 50856 ? S 12:36 0:16 php-fpm: pool customer1 customer1 23635 0.0 0.1 398280 20636 ? S 13:24 0:00 php-fpm: pool customer1 customer1 23636 0.0 0.1 398280 20636 ? S 13:24 0:00 php-fpm: pool customer1 customer1 23637 0.0 0.1 398280 20636 ? S 13:24 0:00 php-fpm: pool customer1 // the one, which is stucked here, is PID 2345 // let's try to kill few processes except 2345 # skill -kill 10887 12018 13157 23635 23637 # ps aux | grep customer1 | grep fpm | wc -l 11 # ps aux | grep customer1 | grep fpm customer1 2345 0.0 0.1 398280 20640 ? S 12:34 0:00 php-fpm: pool customer1 customer1 3295 0.4 0.3 508332 63120 ? S 12:34 0:14 php-fpm: pool customer1 customer1 12707 0.5 0.3 494504 49876 ? S 12:35 0:15 php-fpm: pool customer1 customer1 15860 0.6 0.4 813560 68608 ? S 12:36 0:19 php-fpm: pool customer1 customer1 15941 0.5 0.3 795164 56780 ? R 12:36 0:17 php-fpm: pool customer1 customer1 23636 0.4 0.2 485492 37128 ? R 13:24 0:00 php-fpm: pool customer1 customer1 26450 2.7 0.2 416048 43144 ? S 13:26 0:00 php-fpm: pool customer1 customer1 26451 2.5 0.2 416048 43144 ? S 13:26 0:00 php-fpm: pool customer1 customer1 26452 2.6 0.2 416048 43156 ? S 13:26 0:00 php-fpm: pool customer1 customer1 26453 2.5 0.2 416048 43140 ? S 13:26 0:00 php-fpm: pool customer1 customer1 26454 0.0 0.1 398660 26028 ? S 13:26 0:00 php-fpm: pool customer1 // now let's kill 2345 # skill -kill 2345 # ps aux | grep customer1 | grep fpm | wc -l 1 # ps aux | grep customer1 | grep fpm customer1 27650 0.1 0.1 398660 26760 ? S 13:27 0:00 php-fpm: pool customer1 ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=77023&edit=1