Bug #80739 [Com]: PHP-FPM status page shows listen queue 0

Edit report at https://bugs.php.net/bug.php?id=80739&edit=1 ID: 80739 Comment by: phpbugs at icounsellor dot co dot uk Reported by: micha at dietpi dot com Summary: PHP-FPM status page shows listen queue 0 Status: Open Type: Bug Package: FPM related Operating System: Debian Bullseye PHP Version: 8.0.10 Block user comment: N Private report: N New Comment: Further to Micha's comment: >> I'll now monitor whether "Recv-Q" is ever different from zero. This weekend I wrote a quick-and-dirty set of scripts to see if Micha's plan would show positive results (details below). RESULTS: Day: https://icounsellor.co.uk/phpbugs/phpfpm_queue-day.png Week: https://icounsellor.co.uk/phpbugs/phpfpm_queue-week.png Month: https://icounsellor.co.uk/phpbugs/phpfpm_queue-month.png (These URLs are soft-linked to my live Munin data) CONCLUSION: Although my PHP-FPM site is of low activity, we do have positive results here showing we can get the value of Receive-Q on Linux RECOMMENDATION: As Micha suggests earlier, the PHP Development Team will be able to gain knowledge from Linux source code (reference: https://serverfault.com/a/930677/577419 - internals of "ss") to attain the "how to" with respect to gaining Receive-Q data for PHP-FPM. -------------------------------- FYI - SCRIPTS. SCRIPT #1: Quick & Dirty Bash Script to collate maximum Receive-Q value for php-fpm7.4. Purpose: Munin runs only once every 5 minutes so asking it to call ss to show receive-q values for php-fpm once every 5 minutes isn't very detailed. Hence, this script runs (... repeatedly via cron) collating, once a second, the maximum value of phpfpm-7.4 Receive-Q via ss (in periods of 5 minutes). Source: #!/bin/bash FileRoot="php7.4-fpm"; Socket=$FileRoot".sock"; LoopDelay=1; SharedFile="/dev/shm/"$FileRoot".count"; # Run for 5 minutes... then quit let RunLoop=300/$LoopDelay; COUNTER=0 Max=0; if [ ! -f $SharedFile ]; then # https://unix.stackexchange.com/questions/107038/obtain-exclusive-read-write-lock-on-a-file-for-atomic-updates flock -x -w 5 $SharedFile echo 0 > $SharedFile; fi while [ $COUNTER -lt $RunLoop ]; do SendQCount=$(ss -ln | grep $Socket | awk '{print $3}'); if (( SendQCount > Max )); then Max=$SendQCount; fi sleep $LoopDelay; let COUNTER=COUNTER+1; done # Write the latest results... flock -x -w 5 $SharedFile echo $Max > $SharedFile; ----- SCRIPT #2: Quick and Dirty Munin plugin Purpose: Munin-node calls this script, reads the data left by the above script (once every 5 minutes), and graphs the information Source: #!/bin/bash if [ "$1" = "config" ]; then MaxQueue=$(ss -ln | grep php7.4-fpm.sock | awk '{print $4}'); echo 'graph_title PHP-FPM Socket Queue Statistics' echo 'graph_vlabel Count' echo 'graph_category php' echo "graph_info This graph counts the number connections queued. Max queue configured to be: "$MaxQueue echo "Current.label Currently Queued Requests"; exit 0 fi CurrentQueue=$(cat /dev/shm/php7.4-fpm.count); echo Current.value $CurrentQueue; exit 0; Previous Comments: ------------------------------------------------------------------------ [2021-09-16 12:27:49] micha at dietpi dot com Indeed SO_LISTENINCQLEN seems to not exist on Linux: https://manpages.debian.org/bullseye/manpages/socket.7.en.html#Socket_options I tried to find out other ways to get current and max queue length on Linux: https://serverfault.com/a/930677/577419 ``` # ss -lx | sed -n '1p;/php/p' Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process u_str LISTEN 0 1024 /run/php/php8.0-fpm.sock 157408202 * 0 ``` If I understood it right, this means current queue length is zero and max is 1024? The latter is true, changing "listen.backlog" to 2048 makes "Send-Q" change accordingly. I'll now monitor whether "Recv-Q" is ever different from zero. Checking the internals of "ss" may then provide the options for the FPM as well, I hope. Best regards, Micha ------------------------------------------------------------------------ [2021-05-28 02:02:23] yuta at adachi dot life I reproduced this as well. Checking the implementation, we see that "getsockopt" is used for getting the queue length. We can use the "SO_LISTENINCQLEN" option in FreeBSD to get the status of the UNIX domain socket, but I think there is no option in Linux to get it. https://github.com/php/php-src/blob/php-8.0.6/sapi/fpm/fpm/fpm_sockets.c#L487-L563 The original implementation did not support UNIX domain sockets, so I suppose this is more of an undocumented specification than a but. http://svn.php.net/viewvc/php/php-src/branches/PHP_5_4/sapi/fpm/fpm/fpm_sockets.c?sortby=date&r1=305266&r2=305267&pathrev=312922& We can use "netlink" to get the status of all sockets, but I think there are some privilege and performance concerns. Any ideas? ------------------------------------------------------------------------ [2021-02-12 15:21:52] micha at dietpi dot com Description: ------------ I enabled the PHP-FPM status page and configured a handler in Apache2: ``` <Location /status> SetHandler "proxy:unix:/run/php/php8.0-fpm.sock|fcgi://localhost/status" </Location> ``` This is the result: ``` pool: www process manager: static start time: 28/Jan/2021:15:53:44 +0100 start since: 1147547 accepted conn: 1054191 listen queue: 0 max listen queue: 0 listen queue len: 0 idle processes: 11 active processes: 1 total processes: 12 max active processes: 12 max children reached: 0 slow requests: 0 ``` As can be seen, while all running processes have been used concurrently (this is the case after a few hours already, so not a very rare event), the listen queue stays at zero. Especially the listen queue len stays at zero, even that listen.backlog = 1024 is applied in the pool configuration and as well system-wide a sufficient backlog is permitted: ``` net.core.somaxconn = 2048 net.ipv4.tcp_max_syn_backlog = 1024 ``` This is on PHP8.0.2, but it was the same before on PHP8.0.1 and PHP7.4. Steps to reproduce the behavior: 1. Install Debian Bullseye 2. Install Apache2 and PHP-FPM 3. Enable the PHP-FPM status page and the related handler in Apache2 4. Cause more concurrent requests than pm.max_children 5. Watch PHP-FPM status page to show listen queue len and max listen queue being zero. Probably related bug report: https://bugs.php.net/bug.php?id=76323 Expected result: ---------------- I would expect that listen queue len matches listen.backlog and that I do see a non-zero value at max listen queue, when the process limit is hit regularly. But probably in this setup it is handled differently? At least connections are not dropped from what I can say, no related error messages appear in either PHP or Apache2 logs, so in fact requests are queried somewhere. Actual result: -------------- listen queue: 0 max listen queue: 0 listen queue len: 0 ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=80739&edit=1