Bug #74709 [Com]: PHP-FPM process eating 100% CPU attempting to use kill_all_lockers
Edit report at https://bugs.php.net/bug.php?id=74709&edit=1
ID: 74709
Comment by: syazov at plesk dot com
Reported by: devel at jasonwoods dot me dot uk
Summary: PHP-FPM process eating 100% CPU attempting to use
kill_all_lockers
Status: Open
Type: Bug
Package: FPM related
Operating System: CentOS 6
PHP Version: 5.6.30
Block user comment: N
Private report: N
New Comment:
There is no infinite loop since https://github.com/php/php-src/commit/bcee2fdbec4f4bba59d4134003cfaf5b1f9b67ab
However, if some pool keeps the lock all other pools fail to process queries because of bailing out
at https://github.com/php/php-src/blob/bcee2fdbec4f4bba59d4134003cfaf5b1f9b67ab/ext/opcache/ZendAccelerator.c#L657.
STR
1. set opcache.force_restart_timeout to X seconds (eg 10sec)
2. create 3 domains with php-fpm handler (sleep.cc, reset.cc, info.cc)
3. create do.php with call of sleep(Y) when Y > X (eg 120sec) in the scope of
sleep.cc
4. create do.php with call of opcache_reset() in the scope of reset.cc
5. create do.php with call of phpinfo() in the scope of info.cc
6. open info.cc/do.php, it should work as expected
7. open sleep.cc/do.php to acquire a long lock of cache
8. open reset.cc/do.php to schedule reset of cache
9. wait X+ seconds and try to open info.cc/do.php
Actual result
pool which serves info.cc tries to kill the process which executes sleep.cc/do.php, failed with
EPERM, info.cc/do.php isn't executed but downloaded
Expected result
info.cc/do.php is executed
I solve it with replacing of ACCEL_LOG_ERROR with ACCEL_LOG_WARNING so all
queries which are received after scheduling of reset and before performing of reset are just
processed without opcache.
Previous Comments:
------------------------------------------------------------------------
[2017-08-17 13:30:44] tarasov dot igor at gmail dot com
So, then this seems to be unrelated. My case should be one of these:
https://bugs.php.net/bug.php?id=61558
https://bugs.php.net/bug.php?id=70185
https://bugs.php.net/bug.php?id=73056
------------------------------------------------------------------------
[2017-08-17 12:25:42] devel at jasonwoods dot me dot uk
tarasov dot igor at gmail dot com
child process
------------------------------------------------------------------------
[2017-08-17 10:12:22] tarasov dot igor at gmail dot com
devel at jasonwoods dot me dot uk, what process was spinning in your case - master or child?
------------------------------------------------------------------------
[2017-08-17 08:42:56] devel at jasonwoods dot me dot uk
Hi tarasov dot igor at gmail dot com
Possibly your issue is unrelated to this one. This issue pertains to kill_all_lockers failing and
triggering a loop within a child process as it attempts to kill opcache lockers for processes
running as other users.
Your issue seems to be related to child death and the parent looping to respawn (or along those
lines) - I suggest raising your issue in a new bug ticket so it can be looked at independently.
------------------------------------------------------------------------
[2017-08-17 07:08:55] tarasov dot igor at gmail dot com
I am experiencing the same issue on Ubuntu 16.04.3 with php 7.0.22 on 8 core system with 16G of RAM.
I have two pools (one with pm = dynamic, and the other is pm = static with single child). I get this
bug constantly, every few hours my server goes into that spinning mode with all dynamic pool
processes getting killed and only master process left with the process from static pool. Master
process takes around 80% of all 8 cores. In the fpm-log I get the following lines:
[16-Aug-2017 20:49:03] NOTICE: [pool www] child 8169 exited with code 0 after 0.034016 seconds from
start
[16-Aug-2017 20:49:03] NOTICE: [pool www] child 8179 started
Repeated about 600 times per second. You can understand that the log grows several GB in size quite
fast and only this log issue can lead to self-produced denial of service once all server space gets
used. I had to reduce log level to warning in order to prevent this.
Reloading FPM service is not helping, I have to restart it in order to fix this behavior.
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=74709
--
Edit this bug report at https://bugs.php.net/bug.php?id=74709&edit=1
Thread (10 messages)