Bug #74709 [Com]: PHP-FPM process eating 100% CPU attempting to use kill_all_lockers
Edit report at https://bugs.php.net/bug.php?id=74709&edit=1
ID: 74709
Comment by: tarasov dot igor at gmail dot com
Reported by: devel at jasonwoods dot me dot uk
Summary: PHP-FPM process eating 100% CPU attempting to use
kill_all_lockers
Status: Open
Type: Bug
Package: FPM related
Operating System: CentOS 6
PHP Version: 5.6.30
Block user comment: N
Private report: N
New Comment:
devel at jasonwoods dot me dot uk, what process was spinning in your case - master or child?
Previous Comments:
------------------------------------------------------------------------
[2017-08-17 08:42:56] devel at jasonwoods dot me dot uk
Hi tarasov dot igor at gmail dot com
Possibly your issue is unrelated to this one. This issue pertains to kill_all_lockers failing and
triggering a loop within a child process as it attempts to kill opcache lockers for processes
running as other users.
Your issue seems to be related to child death and the parent looping to respawn (or along those
lines) - I suggest raising your issue in a new bug ticket so it can be looked at independently.
------------------------------------------------------------------------
[2017-08-17 07:08:55] tarasov dot igor at gmail dot com
I am experiencing the same issue on Ubuntu 16.04.3 with php 7.0.22 on 8 core system with 16G of RAM.
I have two pools (one with pm = dynamic, and the other is pm = static with single child). I get this
bug constantly, every few hours my server goes into that spinning mode with all dynamic pool
processes getting killed and only master process left with the process from static pool. Master
process takes around 80% of all 8 cores. In the fpm-log I get the following lines:
[16-Aug-2017 20:49:03] NOTICE: [pool www] child 8169 exited with code 0 after 0.034016 seconds from
start
[16-Aug-2017 20:49:03] NOTICE: [pool www] child 8179 started
Repeated about 600 times per second. You can understand that the log grows several GB in size quite
fast and only this log issue can lead to self-produced denial of service once all server space gets
used. I had to reduce log level to warning in order to prevent this.
Reloading FPM service is not helping, I have to restart it in order to fix this behavior.
------------------------------------------------------------------------
[2017-08-17 04:41:29] josh at jmarshall dot com dot au
I spoke too soon. Running 5.6.31 just had the problem again. strace shows:
kill(16099, SIGKILL) = -1 EPERM (Operation not permitted)
fcntl(90, F_GETLK, {type=F_RDLCK, whence=SEEK_SET, start=1, len=1, pid=16099}) = 0
------------------------------------------------------------------------
[2017-08-11 02:10:19] josh at jmarshall dot com dot au
I have not noticed this happen since moving to 5.6.31 on my servers, has been 6 weeks now. With
5.6.30 was happening every few days.
Hopefully that is the case for the others experiencing this.
------------------------------------------------------------------------
[2017-06-11 10:19:34] devel at jasonwoods dot me dot uk
Having looked into a few server, it seems the same file is used for all FPM processes. So if there
is any issue with the lock file and kill_all_lockers comes into action, there's a good chance
on most multi-pool servers of the process to start spinning.
Ideally:
- Lock file should be opened after fork in FPM, or maybe even defer extension activation until after
fork, in any case, sharing things amongst the same pool rather than the entire PHP-FPM pool set.
- A backoff on the kill so it performs once every X milliseconds instead of spinning.
I guess I have some concern on the latter though, in that it will fix the CPU but it will then have
the side effect of hiding the fact that a process is effectively deadlocked and the pool-size
reduced. So I wonder if anyone knows if the former, or similar solution, is viable?
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=74709
--
Edit this bug report at https://bugs.php.net/bug.php?id=74709&edit=1
Thread (10 messages)