Bug #80739 [Com]: PHP-FPM status page shows listen queue 0

Edit report at https://bugs.php.net/bug.php?id=80739&edit=1 ID: 80739 Comment by: micha at dietpi dot com Reported by: micha at dietpi dot com Summary: PHP-FPM status page shows listen queue 0 Status: Open Type: Bug Package: FPM related Operating System: Debian Bullseye PHP Version: 8.0.10 Block user comment: N Private report: N New Comment: Many thanks for providing the scripts. Indeed I can also verify with PHP8.0 that this method works fine and "Recv-Q" indeed shows the current query length. I did monitoring with this one-liner from shell: ``` while sleep 1; do value=$(ss -lxn | mawk '/php/{print $3}'); (( $value )) && echo "$value"; done ``` However, this is of course everything else than precise as it only takes a snapshot of the current query length at defined interval, which, considering the speed in which PHP requests are handled, will unlikely ever show the real maximum reached. For the current query length ("listen queue:" on status page) this however works fine. Of course PHP-FPM shouldn't execute any external command but try to obtain the information with the methods "ss" uses ;). I wonder whether /proc/<PID>/net/netstat contains what we are looking for. But so far I couldn't identify any of the long value list a maximum reached query length. Probably there are other stats provided by the kernel in /sys or /proc somewhere? Best regards, Micha Previous Comments: ------------------------------------------------------------------------ [2021-11-08 13:45:01] phpbugs at icounsellor dot co dot uk Further to Micha's comment: >> I'll now monitor whether "Recv-Q" is ever different from zero. This weekend I wrote a quick-and-dirty set of scripts to see if Micha's plan would show positive results (details below). RESULTS: Day: https://icounsellor.co.uk/phpbugs/phpfpm_queue-day.png Week: https://icounsellor.co.uk/phpbugs/phpfpm_queue-week.png Month: https://icounsellor.co.uk/phpbugs/phpfpm_queue-month.png (These URLs are soft-linked to my live Munin data) CONCLUSION: Although my PHP-FPM site is of low activity, we do have positive results here showing we can get the value of Receive-Q on Linux RECOMMENDATION: As Micha suggests earlier, the PHP Development Team will be able to gain knowledge from Linux source code (reference: https://serverfault.com/a/930677/577419 - internals of "ss") to attain the "how to" with respect to gaining Receive-Q data for PHP-FPM. -------------------------------- FYI - SCRIPTS. SCRIPT #1: Quick & Dirty Bash Script to collate maximum Receive-Q value for php-fpm7.4. Purpose: Munin runs only once every 5 minutes so asking it to call ss to show receive-q values for php-fpm once every 5 minutes isn't very detailed. Hence, this script runs (... repeatedly via cron) collating, once a second, the maximum value of phpfpm-7.4 Receive-Q via ss (in periods of 5 minutes). Source: #!/bin/bash FileRoot="php7.4-fpm"; Socket=$FileRoot".sock"; LoopDelay=1; SharedFile="/dev/shm/"$FileRoot".count"; # Run for 5 minutes... then quit let RunLoop=300/$LoopDelay; COUNTER=0 Max=0; if [ ! -f $SharedFile ]; then # https://unix.stackexchange.com/questions/107038/obtain-exclusive-read-write-lock-on-a-file-for-atomic-updates flock -x -w 5 $SharedFile echo 0 > $SharedFile; fi while [ $COUNTER -lt $RunLoop ]; do SendQCount=$(ss -ln | grep $Socket | awk '{print $3}'); if (( SendQCount > Max )); then Max=$SendQCount; fi sleep $LoopDelay; let COUNTER=COUNTER+1; done # Write the latest results... flock -x -w 5 $SharedFile echo $Max > $SharedFile; ----- SCRIPT #2: Quick and Dirty Munin plugin Purpose: Munin-node calls this script, reads the data left by the above script (once every 5 minutes), and graphs the information Source: #!/bin/bash if [ "$1" = "config" ]; then MaxQueue=$(ss -ln | grep php7.4-fpm.sock | awk '{print $4}'); echo 'graph_title PHP-FPM Socket Queue Statistics' echo 'graph_vlabel Count' echo 'graph_category php' echo "graph_info This graph counts the number connections queued. Max queue configured to be: "$MaxQueue echo "Current.label Currently Queued Requests"; exit 0 fi CurrentQueue=$(cat /dev/shm/php7.4-fpm.count); echo Current.value $CurrentQueue; exit 0; ------------------------------------------------------------------------ [2021-09-16 12:27:49] micha at dietpi dot com Indeed SO_LISTENINCQLEN seems to not exist on Linux: https://manpages.debian.org/bullseye/manpages/socket.7.en.html#Socket_options I tried to find out other ways to get current and max queue length on Linux: https://serverfault.com/a/930677/577419 ``` # ss -lx | sed -n '1p;/php/p' Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process u_str LISTEN 0 1024 /run/php/php8.0-fpm.sock 157408202 * 0 ``` If I understood it right, this means current queue length is zero and max is 1024? The latter is true, changing "listen.backlog" to 2048 makes "Send-Q" change accordingly. I'll now monitor whether "Recv-Q" is ever different from zero. Checking the internals of "ss" may then provide the options for the FPM as well, I hope. Best regards, Micha ------------------------------------------------------------------------ [2021-05-28 02:02:23] yuta at adachi dot life I reproduced this as well. Checking the implementation, we see that "getsockopt" is used for getting the queue length. We can use the "SO_LISTENINCQLEN" option in FreeBSD to get the status of the UNIX domain socket, but I think there is no option in Linux to get it. https://github.com/php/php-src/blob/php-8.0.6/sapi/fpm/fpm/fpm_sockets.c#L487-L563 The original implementation did not support UNIX domain sockets, so I suppose this is more of an undocumented specification than a but. http://svn.php.net/viewvc/php/php-src/branches/PHP_5_4/sapi/fpm/fpm/fpm_sockets.c?sortby=date&r1=305266&r2=305267&pathrev=312922& We can use "netlink" to get the status of all sockets, but I think there are some privilege and performance concerns. Any ideas? ------------------------------------------------------------------------ [2021-02-12 15:21:52] micha at dietpi dot com Description: ------------ I enabled the PHP-FPM status page and configured a handler in Apache2: ``` <Location /status> SetHandler "proxy:unix:/run/php/php8.0-fpm.sock|fcgi://localhost/status" </Location> ``` This is the result: ``` pool: www process manager: static start time: 28/Jan/2021:15:53:44 +0100 start since: 1147547 accepted conn: 1054191 listen queue: 0 max listen queue: 0 listen queue len: 0 idle processes: 11 active processes: 1 total processes: 12 max active processes: 12 max children reached: 0 slow requests: 0 ``` As can be seen, while all running processes have been used concurrently (this is the case after a few hours already, so not a very rare event), the listen queue stays at zero. Especially the listen queue len stays at zero, even that listen.backlog = 1024 is applied in the pool configuration and as well system-wide a sufficient backlog is permitted: ``` net.core.somaxconn = 2048 net.ipv4.tcp_max_syn_backlog = 1024 ``` This is on PHP8.0.2, but it was the same before on PHP8.0.1 and PHP7.4. Steps to reproduce the behavior: 1. Install Debian Bullseye 2. Install Apache2 and PHP-FPM 3. Enable the PHP-FPM status page and the related handler in Apache2 4. Cause more concurrent requests than pm.max_children 5. Watch PHP-FPM status page to show listen queue len and max listen queue being zero. Probably related bug report: https://bugs.php.net/bug.php?id=76323 Expected result: ---------------- I would expect that listen queue len matches listen.backlog and that I do see a non-zero value at max listen queue, when the process limit is hit regularly. But probably in this setup it is handled differently? At least connections are not dropped from what I can say, no related error messages appear in either PHP or Apache2 logs, so in fact requests are queried somewhere. Actual result: -------------- listen queue: 0 max listen queue: 0 listen queue len: 0 ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=80739&edit=1