From: tozz at kijkt dot tv
Operating system: Debian 9
PHP version: 7.0.32
Package: FPM related
Bug Type: Bug
Bug description:FPM occassionally hangs
Description:
------------
We sometimes face the issue that the FPM-daemon of PHP no longer
responds to any queries/hits/requests. We've seen this issue occur on
multiple on our servers, and it tends to happen to multiple servers over
the course of a few days (2/3 days) and afterwards it can be fine for
weeks. So it seems there is something happening on these servers that
triggers it, but I have no clue what it is.
What happens is:
- All requests for sites using this FPM hang indefinitly (timeouts are
seemingly ignored)
- After a while, Apache has 0 free slots and goes down as well (due to
not having available slots)
- Restarting Apache solves the slot issue, but FPM is still not working,
so slots drop to zero after a while again.
- Restarting FPM instantly resolves all issues (all busy Apache slots
are freed, everything starts working again).
I suspect this has something to do with frequent restarts of the FPM
daemon (due to external configuration changes), but cannot be certain.
There are no issues in the FPM logfile.
When I strace the process I see:
strace: Process 1217 attached
getpid() = 1217
epoll_wait(26, [], 381, 90) = 0
getpid() = 1217
getpid() = 1217
getpid() = 1217
epoll_wait(26, [], 381, 1000) = 0
getpid() = 1217
getpid() = 1217
getpid() = 1217
epoll_wait(26, [], 381, 1000) = 0
getpid() = 1217
getpid() = 1217
getpid() = 1217
epoll_wait(26, [], 381, 1000) = 0
getpid() = 1217
getpid() = 1217
getpid() = 1217
epoll_wait(26, [], 381, 1000) = 0
getpid() = 1217
getpid() = 1217
getpid() = 1217
epoll_wait(26, [], 381, 1000) = 0
getpid() = 1217
getpid() = 1217
getpid() = 1217
epoll_wait(26, [], 381, 1000) = 0
getpid() = 1217
getpid() = 1217
getpid() = 1217
epoll_wait(26, [], 381, 238) = 0
getpid() = 1217
socket(AF_UNIX, SOCK_DGRAM|SOCK_CLOEXEC, 0) = 27
getsockopt(27, SOL_SOCKET, SO_SNDBUF, [212992], [4]) = 0
setsockopt(27, SOL_SOCKET, SO_SNDBUFFORCE, [8388608], 4) = 0
sendmsg(27, {msg_name={sa_family=AF_UNIX,
sun_path="/run/systemd/notify"}, msg_namelen=21,
msg_iov=[{iov_base="READY=1\nSTATUS=Processes active:"..., iov_len=87}],
msg_iovlen=1, msg_controllen=0, msg_flags=0}, MSG_NOSIGNAL) = 87
close(27) = 0
getpid() = 1217
getpid() = 1217
epoll_wait(26, [], 381, 761) = 0
getpid() = 1217
getpid() = 1217
getpid() = 1217
epoll_wait(26, [], 381, 1000) = 0
getpid() = 1217
getpid() = 1217
getpid() = 1217
epoll_wait(26, [], 381, 1000) = 0
getpid() = 1217
getpid() = 1217
getpid() = 1217
epoll_wait(26, [], 381, 1000) = 0
getpid() = 1217
getpid() = 1217
getpid() = 1217
epoll_wait(26, ^Cstrace: Process 1217 detached
<detached ...>
I have no experience with 'gdb', so I don't really know what to do, but
I tried to get a trace. This didn't look very usefull to me, but who
knows:
(gdb) bt
#0 0x00007f82247150a3 in __epoll_wait_nocancel () at
../sysdeps/unix/syscall-template.S:84
#1 0x000055f6ceccfc43 in ?? ()
#2 0x000055f6cecc6144 in fpm_event_loop ()
#3 0x000055f6cecc0b37 in fpm_run ()
#4 0x000055f6cea3bc6b in main ()
(gdb)
(gdb) info threads
Id Target Id Frame
* 1 Thread 0x7f82277a58c0 (LWP 1217) "php-fpm" 0x00007f82247150a3 in
__epoll_wait_nocancel () at ../sysdeps/unix/syscall-template.S:84
(gdb)
I tried to replicate the issue, but failed to do so. I also have limited
time to debug this issue, because it is causing downtime.
We also see this issue on multiple PHP versions (ranging from 5.6 to
7.2).
--
Edit bug report at https://bugs.php.net/bug.php?id=77140&edit=1
--
Try a snapshot (PHP 5.4): https://bugs.php.net/fix.php?id=77140&r=trysnapshot54
Try a snapshot (PHP 5.5): https://bugs.php.net/fix.php?id=77140&r=trysnapshot55
Try a snapshot (trunk): https://bugs.php.net/fix.php?id=77140&r=trysnapshottrunk
Fixed in SVN: https://bugs.php.net/fix.php?id=77140&r=fixed
Fixed in release: https://bugs.php.net/fix.php?id=77140&r=alreadyfixed
Need backtrace: https://bugs.php.net/fix.php?id=77140&r=needtrace
Need Reproduce Script: https://bugs.php.net/fix.php?id=77140&r=needscript
Try newer version: https://bugs.php.net/fix.php?id=77140&r=oldversion
Not developer issue: https://bugs.php.net/fix.php?id=77140&r=support
Expected behavior: https://bugs.php.net/fix.php?id=77140&r=notwrong
Not enough info: https://bugs.php.net/fix.php?id=77140&r=notenoughinfo
Submitted twice: https://bugs.php.net/fix.php?id=77140&r=submittedtwice
register_globals: https://bugs.php.net/fix.php?id=77140&r=globals
PHP 4 support discontinued: https://bugs.php.net/fix.php?id=77140&r=php4
Daylight Savings: https://bugs.php.net/fix.php?id=77140&r=dst
IIS Stability: https://bugs.php.net/fix.php?id=77140&r=isapi
Install GNU Sed: https://bugs.php.net/fix.php?id=77140&r=gnused
Floating point limitations: https://bugs.php.net/fix.php?id=77140&r=float
No Zend Extensions: https://bugs.php.net/fix.php?id=77140&r=nozend
MySQL Configuration Error: https://bugs.php.net/fix.php?id=77140&r=mysqlcfg