[php-src] Issue #14149: PHP-FPM requests are stuck and childs are killed after elapsing of the timeout

From: Date: Mon, 06 May 2024 13:48:04 +0000
Subject: [php-src] Issue #14149: PHP-FPM requests are stuck and childs are killed after elapsing of the timeout
Groups: php.bugs 
Request: Send a blank email to php-bugs+get-246863@lists.php.net to get a copy of this message
Issue: https://github.com/php/php-src/issues/14149 Author: WoZ ### Description php-fpm child processes periodically stuck. No matter what scripts are executing... Script may stuck and the same script with the same params will execute ok right after this. My application has logs and writes them to stdout/stderr, but during the mentioned "stucks" no logs are present at all, so the logger didn't initialize, and initialization is the first step, even before any external connections to databases/queues/caches. Looks like a bug. I use a [base docker image 8.1.15-fpm](https://hub.docker.com/layers/library/php/8.1.15-fpm/images/sha256-1de3854347696a1a1ca6ec2b1f8ebc146cf17aa9e0088c90eb5f408492e5106a?context=explore). nginx is used as a webserver with FastCGI support and nginx routes via upstream requests to the same host machine (different container) via TCP. Configuration: ```ini [global] error_log = /proc/self/fd/2 log_level = debug log_limit = 8192 daemonize = no [www] access.log = /proc/self/fd/2 slowlog = /proc/self/fd/2 request_slowlog_timeout = 60s clear_env = no catch_workers_output = yes decorate_workers_output = no pm = dynamic pm.max_children = 50 pm.start_servers = 2 pm.min_spare_servers = 2 pm.max_spare_servers = 5 pm.status_path = /status request_terminate_timeout = 60s listen = 9000 ``` Additional info and logs during such a stuck. 1. nginx forwards a request to the upstream (php-fpm) on the same machine, but in a different container. 2. php-fpm logs nothing related to this request and this child. Only the very first message from the master process regarding ``` [02-May-2024 12:27:00.660632] DEBUG: pid 7, fpm_children_make(), line 435: [pool www] child 215893 started. ``` 4. I write every second an output of the status page to the log and may see this request. ``` { "pool": "www", "process manager": "dynamic", "start time": 1714984262, "start since": 14215, "accepted conn": 71133, "listen queue": 0, "max listen queue": 1, "listen queue len": 511, "idle processes": 3, "active processes": 3, "total processes": 6, "max active processes": 16, "max children reached": 0, "slow requests": 0, "processes": [ { "pid": 215893, "state": "Running", "start time": 1714998420, "start since": 57, "requests": 15, "request duration": 35713306, "request method": "POST", "request uri": "/index.php?access-token=secret_key_1&match_id=1111823&period_id=&event_code=", "content length": 95, "user": "-", "script": "/api/public/index.php", "last request cpu": 0, "last request memory": 0 }, { "pid": 215774, "state": "Idle", "start time": 1714998270, "start since": 207, "requests": 188, "request duration": 77764, "request method": "-", "request uri": "-", "content length": 0, "user": "-", "script": "-", "last request cpu": 51.44, "last request memory": 8388608 }, { "pid": 215799, "state": "Idle", "start time": 1714998303, "start since": 174, "requests": 164, "request duration": 5309, "request method": "-", "request uri": "-", "content length": 0, "user": "-", "script": "-", "last request cpu": 0, "last request memory": 16777216 }, { "pid": 215766, "state": "Idle", "start time": 1714998267, "start since": 210, "requests": 194, "request duration": 66407, "request method": "-", "request uri": "-", "content length": 0, "user": "-", "script": "-", "last request cpu": 30.12, "last request memory": 8388608 }, { "pid": 209262, "state": "Finishing", "start time": 1714990534, "start since": 7943, "requests": 7, "request duration": 7939544520, "request method": "POST", "request uri": "/index.php?access-token=secret_key_1&match_id=1112831&period_id=&event_code=", "content length": 92, "user": "-", "script": "/api/public/index.php", "last request cpu": 0, "last request memory": 0 }, { "pid": 215484, "state": "Running", "start time": 1714997843, "start since": 634, "requests": 605, "request duration": 4025, "request method": "GET", "request uri": "/status?full&json", "content length": 0, "user": "-", "script": "-", "last request cpu": 0, "last request memory": 0 } ] } ``` 5. After some time php-fpm kills this child with the message in the log file ``` [02-May-2024 12:28:23.221826] WARNING: pid 7, fpm_request_check_timed_out(), line 272: [pool www] child 215893, script '/api/public/index.php' (request: "POST /index.php?access-token=secret_key_1&match_id=1111823&period_id=&event_code=") execution timed out (61.052097 sec), terminating [02-May-2024 12:28:23.228318] WARNING: pid 7, fpm_children_bury(), line 258: [pool www] child 215893 exited on signal 15 (SIGTERM) after 82.568327 seconds from start ``` 6. The application didn't send anything to nginx, because in nginx logs I see ``` 172.0.1.10 - - [06/May/2024:12:27:58 +0000] "POST /action?access-token=secret_key_1&match_id=1111823&period_id=&event_code= HTTP/1.0" 499 0 "https://origdomain.com/" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36" "XXX.XXX.XXX.XXX, XXX.XXX.XXX.XXX, XXX.XXX.XXX.XXX" rt="36.043" uct="0.000" uht="-" urt="36.044" conn="1286328" conn_reqs="1" upstream_addr="172.0.1.6:9000" upstream_status="-" upstream_response_time="36.044" upstream_header_time="-" upstream_connect_time="0.000" upstream_cache_status="-" upstream_bytes_sent="1888" upstream_bytes_received="0" ``` HTTP status code is 499, because nginx closed the connection due to max wait time. upstream_connect_time="0.000", upstream_bytes_sent="1888" upstream_bytes_received="0". There is no pattern present. Simple health checks fail as well as more complicated logic. The only thing in common is that "requests" for the affected child are always low, and the child starts <60s before the stuck. Any ideas? Paid help for investigation and resolution is welcomed too. ### PHP Version 8.1.15 ### Operating System Ubuntu 20.04 in docker

« previous php.bugs (#246863) next »