[php-src] Issue #11447: fpm: some workers get stuck on "\0fscf\0" write to master process over time

From: Date: Tue, 13 Jun 2023 11:31:58 +0000
Subject: [php-src] Issue #11447: fpm: some workers get stuck on "\0fscf\0" write to master process over time
Groups: php.bugs 
Request: Send a blank email to php-bugs+get-244682@lists.php.net to get a copy of this message
Issue: https://github.com/php/php-src/issues/11447 Author: thias ### Description After updating from PHP 7.3 to 8.1, we are now seeing the number of "active" php-fpm processes slowly grow over time, which is not expected. Looking at the fpm-status output, many processes report a duration of multiple days. name | value -- | -- pool | www1 process manager | static start time | 6/5/2023, 11:25:15 PM start since | 7d 13h 181 accepted conn | 1109651666 listen queue | 0 max listen queue | 0 listen queue len | 0 idle processes | 678 active processes | 90 total processes | 768 max active processes | 768 max children reached | 0 slow requests | 466772 In the above, after restarting php-fpm, the number of active processes goes back down to around 60. So only about 30 got stuck out of 1 billion accepted connections. Really not that many, but resulting in +50% active processes after a week, and getting worse as time goes. Looking at the stuck processes, all are on the exact same write. Here's one example: ``` # strace -p 5843 strace: Process 5843 attached write(2, "\0fscf\0", 6 ``` That fd is a pipe to the master php-fpm process: ``` php-fpm 5843 phpwww 2w FIFO 0,13 0t0 2303333681 pipe ``` ``` php-fpm 5843 phpwww 2w FIFO 0,13 0t0 2303333681 pipe php-fpm 91227 root 1392r FIFO 0,13 0t0 2303333681 pipe ``` ``` 91227 ? Ss 339:39 php-fpm: master process (/etc/php-fpm1.conf) 79811 ? S 1:37 \_ php-fpm: pool www1 80146 ? S 1:38 \_ php-fpm: pool www1 80914 ? S 1:36 \_ php-fpm: pool www1 [...] ``` The fscf string can be found in sapi/fpm/fpm/fpm_stdio.c: ``` #define FPM_STDIO_CMD_FLUSH "\0fscf" ``` So for some reason, it looks like in some rare cases, sending this command from a worker back to the master gets stuck forever. ### PHP Version PHP 8.1.20 ### Operating System RHEL 7.9

« previous php.bugs (#244682) next »