[php-src] Issue #14149: PHP-FPM requests are stuck and childs are killed after elapsing of the timeout
| From: | WoZ | Date: | Mon, 06 May 2024 13:48:04 +0000 |
| Subject: | [php-src] Issue #14149: PHP-FPM requests are stuck and childs are killed after elapsing of the timeout | ||
| Groups: | php.bugs | ||
| Request: | Send a blank email to php-bugs+get-246863@lists.php.net to get a copy of this message | ||
Issue: https://github.com/php/php-src/issues/14149
Author: WoZ
### Description
php-fpm child processes periodically stuck. No matter what scripts are executing... Script may stuck
and the same script with the same params will execute ok right after this. My application has logs
and writes them to stdout/stderr, but during the mentioned "stucks" no logs are present at
all, so the logger didn't initialize, and initialization is the first step, even before any
external connections to databases/queues/caches. Looks like a bug.
I use a [base docker image
8.1.15-fpm](https://hub.docker.com/layers/library/php/8.1.15-fpm/images/sha256-1de3854347696a1a1ca6ec2b1f8ebc146cf17aa9e0088c90eb5f408492e5106a?context=explore).
nginx is used as a webserver with FastCGI support and nginx routes via upstream requests to the same
host machine (different container) via TCP.
Configuration:
```ini
[global]
error_log = /proc/self/fd/2
log_level = debug
log_limit = 8192
daemonize = no
[www]
access.log = /proc/self/fd/2
slowlog = /proc/self/fd/2
request_slowlog_timeout = 60s
clear_env = no
catch_workers_output = yes
decorate_workers_output = no
pm = dynamic
pm.max_children = 50
pm.start_servers = 2
pm.min_spare_servers = 2
pm.max_spare_servers = 5
pm.status_path = /status
request_terminate_timeout = 60s
listen = 9000
```
Additional info and logs during such a stuck.
1. nginx forwards a request to the upstream (php-fpm) on the same machine, but in a different
container.
2. php-fpm logs nothing related to this request and this child. Only the very first message from the
master process regarding
```
[02-May-2024 12:27:00.660632] DEBUG: pid 7, fpm_children_make(), line 435: [pool www] child 215893
started.
```
4. I write every second an output of the status page to the log and may see this request.
```
{
"pool": "www",
"process manager": "dynamic",
"start time": 1714984262,
"start since": 14215,
"accepted conn": 71133,
"listen queue": 0,
"max listen queue": 1,
"listen queue len": 511,
"idle processes": 3,
"active processes": 3,
"total processes": 6,
"max active processes": 16,
"max children reached": 0,
"slow requests": 0,
"processes": [
{
"pid": 215893,
"state": "Running",
"start time": 1714998420,
"start since": 57,
"requests": 15,
"request duration": 35713306,
"request method": "POST",
"request uri":
"/index.php?access-token=secret_key_1&match_id=1111823&period_id=&event_code=",
"content length": 95,
"user": "-",
"script": "/api/public/index.php",
"last request cpu": 0,
"last request memory": 0
},
{
"pid": 215774,
"state": "Idle",
"start time": 1714998270,
"start since": 207,
"requests": 188,
"request duration": 77764,
"request method": "-",
"request uri": "-",
"content length": 0,
"user": "-",
"script": "-",
"last request cpu": 51.44,
"last request memory": 8388608
},
{
"pid": 215799,
"state": "Idle",
"start time": 1714998303,
"start since": 174,
"requests": 164,
"request duration": 5309,
"request method": "-",
"request uri": "-",
"content length": 0,
"user": "-",
"script": "-",
"last request cpu": 0,
"last request memory": 16777216
},
{
"pid": 215766,
"state": "Idle",
"start time": 1714998267,
"start since": 210,
"requests": 194,
"request duration": 66407,
"request method": "-",
"request uri": "-",
"content length": 0,
"user": "-",
"script": "-",
"last request cpu": 30.12,
"last request memory": 8388608
},
{
"pid": 209262,
"state": "Finishing",
"start time": 1714990534,
"start since": 7943,
"requests": 7,
"request duration": 7939544520,
"request method": "POST",
"request uri":
"/index.php?access-token=secret_key_1&match_id=1112831&period_id=&event_code=",
"content length": 92,
"user": "-",
"script": "/api/public/index.php",
"last request cpu": 0,
"last request memory": 0
},
{
"pid": 215484,
"state": "Running",
"start time": 1714997843,
"start since": 634,
"requests": 605,
"request duration": 4025,
"request method": "GET",
"request uri": "/status?full&json",
"content length": 0,
"user": "-",
"script": "-",
"last request cpu": 0,
"last request memory": 0
}
]
}
```
5. After some time php-fpm kills this child with the message in the log file
```
[02-May-2024 12:28:23.221826] WARNING: pid 7, fpm_request_check_timed_out(), line 272: [pool www]
child 215893, script '/api/public/index.php' (request: "POST
/index.php?access-token=secret_key_1&match_id=1111823&period_id=&event_code=")
execution timed out (61.052097 sec), terminating
[02-May-2024 12:28:23.228318] WARNING: pid 7, fpm_children_bury(), line 258: [pool www] child 215893
exited on signal 15 (SIGTERM) after 82.568327 seconds from start
```
6. The application didn't send anything to nginx, because in nginx logs I see
```
172.0.1.10 - - [06/May/2024:12:27:58 +0000] "POST
/action?access-token=secret_key_1&match_id=1111823&period_id=&event_code= HTTP/1.0"
499 0 "https://origdomain.com/"
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/124.0.0.0 Safari/537.36" "XXX.XXX.XXX.XXX, XXX.XXX.XXX.XXX, XXX.XXX.XXX.XXX"
rt="36.043" uct="0.000" uht="-" urt="36.044"
conn="1286328" conn_reqs="1" upstream_addr="172.0.1.6:9000"
upstream_status="-" upstream_response_time="36.044"
upstream_header_time="-" upstream_connect_time="0.000"
upstream_cache_status="-" upstream_bytes_sent="1888"
upstream_bytes_received="0"
```
HTTP status code is 499, because nginx closed the connection due to max wait time.
upstream_connect_time="0.000", upstream_bytes_sent="1888"
upstream_bytes_received="0".
There is no pattern present. Simple health checks fail as well as more complicated logic. The only
thing in common is that "requests" for the affected child are always low, and the child
starts <60s before the stuck.
Any ideas?
Paid help for investigation and resolution is welcomed too.
### PHP Version
8.1.15
### Operating System
Ubuntu 20.04 in docker