[php-src] Issue #11447: fpm: some workers get stuck on "\0fscf\0" write to master process over time
| From: | thias | Date: | Tue, 13 Jun 2023 11:31:58 +0000 |
| Subject: | [php-src] Issue #11447: fpm: some workers get stuck on "\0fscf\0" write to master process over time | ||
| Groups: | php.bugs | ||
| Request: | Send a blank email to php-bugs+get-244682@lists.php.net to get a copy of this message | ||
Issue: https://github.com/php/php-src/issues/11447
Author: thias
### Description
After updating from PHP 7.3 to 8.1, we are now seeing the number of "active" php-fpm
processes slowly grow over time, which is not expected. Looking at the fpm-status output, many
processes report a duration of multiple days.
name | value
-- | --
pool | www1
process manager | static
start time | 6/5/2023, 11:25:15 PM
start since | 7d 13h 181
accepted conn | 1109651666
listen queue | 0
max listen queue | 0
listen queue len | 0
idle processes | 678
active processes | 90
total processes | 768
max active processes | 768
max children reached | 0
slow requests | 466772
In the above, after restarting php-fpm, the number of active processes goes back down to around 60.
So only about 30 got stuck out of 1 billion accepted connections. Really not that many, but
resulting in +50% active processes after a week, and getting worse as time goes.
Looking at the stuck processes, all are on the exact same write. Here's one example:
```
# strace -p 5843
strace: Process 5843 attached
write(2, "\0fscf\0", 6
```
That fd is a pipe to the master php-fpm process:
```
php-fpm 5843 phpwww 2w FIFO 0,13 0t0 2303333681 pipe
```
```
php-fpm 5843 phpwww 2w FIFO 0,13 0t0 2303333681 pipe
php-fpm 91227 root 1392r FIFO 0,13 0t0 2303333681 pipe
```
```
91227 ? Ss 339:39 php-fpm: master process (/etc/php-fpm1.conf)
79811 ? S 1:37 \_ php-fpm: pool www1
80146 ? S 1:38 \_ php-fpm: pool www1
80914 ? S 1:36 \_ php-fpm: pool www1
[...]
```
The fscf string can be found in sapi/fpm/fpm/fpm_stdio.c:
```
#define FPM_STDIO_CMD_FLUSH "\0fscf"
```
So for some reason, it looks like in some rare cases, sending this command from a worker back to the
master gets stuck forever.
### PHP Version
PHP 8.1.20
### Operating System
RHEL 7.9