Edit report at https://bugs.php.net/bug.php?id=61558&edit=1
ID: 61558
Updated by: bukka@php.net
Reported by: phpbug2012 at tgabber dot mine dot nu
Summary: Runaway spawning of children after pipe error
-Status: Analyzed
+Status: Assigned
Type: Bug
Package: FPM related
Operating System: Debian Linux
PHP Version: 5.3.10
-Assigned To:
+Assigned To: bukka
Block user comment: N
Private report: N
New Comment:
The actual issue was resolved a long time ago. We kept it open just as a reminder that we should do
something about handling of a quick child crash in FPM master to not become unresponsive. I thought
more about implementation of that and write it down to https://github.com/php/php-src/issues/9632 so
closing this.
Previous Comments:
------------------------------------------------------------------------
[2015-08-03 03:25:28] kobenews at cox dot net
I think this issue is related to a bug I just posted. Same behavior where php-fpm restarts the
process over and over again, causing php-fpm master to use 100% cpu. The test script is a little bit
simpler, though requires exec() and running a SSH command using multiplexing.
https://bugs.php.net/bug.php?id=70185&edit=2
------------------------------------------------------------------------
[2014-07-25 13:07:22] Danack at basereality dot com
I'd recommend being inspired by aka copying the behaviour of Supervisord.
http://supervisord.org/subprocess.html
Basically it does what it probably the best behaviour of:
i) If child processes exist too quickly, put them in a 'back off' state, to rate limit the
number of processes attempting to be started.
ii) If they still fail to start after a reasonable number of retries stop trying to start it for
now.
Admittedly, sounds like a big task for something that should be a rare event.
------------------------------------------------------------------------
[2014-07-25 09:06:12] tony2001@php.net
Ok, so FPM basically starts the children, but they die immediately (for a reason) and FPM master
process starts the new ones.
I can think of adding some option that would limit the number of children spawned per second, or
probably abort the whole process if N children were created in the last N seconds, but that kind of
solution looks quite hacky to me.
------------------------------------------------------------------------
[2014-07-24 19:44:00] Danack at basereality dot com
I think there is an easier way to trigger this (and probably something that I'll open as a
separate bug):
i) Include a library function in an extension.
ii) Don't include the library in the linking step for php-fpm.
iii) Watch the error log fill up with:
[24-Jul-2014 18:58:27] NOTICE: [pool www] child 22710 started
[24-Jul-2014 18:58:27] WARNING: [pool www] child 22710 exited with code 127 after 0.044963 seconds
from start
[24-Jul-2014 18:58:27] WARNING: [pool www] child 22710 said into stderr: "php-fpm: pool www:
symbol lookup error: /usr/local/lib/php/extensions/no-debug-zts-20131226/imagick.so: undefined
symbol: GetMagickVersion"
Once this happens you need to kill -9 the php-fpm master process to stop it spawning pool workers.
Obviously there shouldn't be errors like this in an extension, but also PHP-FPM shouldn't
become unresponsive and start spawning workers like crazy.
------------------------------------------------------------------------
[2013-05-23 13:44:24] pavel at stack dot ee
PHP 5.3.23 with Suhosin-Patch (cli) (built: Mar 26 2013 14:07:09)
Copyright (c) 1997-2013 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2013 Zend Technologies
FPM POOL CONFIG:
[storage]
listen = /tmp/fpm_storage.sock
listen.owner = storage
listen.group = storage
listen.mode = 0666
user = storage
group = storage
pm = dynamic
pm.max_children = 50
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 35
NGINX:
location ~ \.php$ {
fastcgi_pass unix:/tmp/fpm_storage.sock;
fastcgi_index index.php;
include fastcgi_params;
fastcgi_param DEVENV on;
fastcgi_intercept_errors on;
fastcgi_ignore_client_abort off;
fastcgi_connect_timeout 60;
fastcgi_send_timeout 180;
fastcgi_read_timeout 180;
fastcgi_buffer_size 128k;
fastcgi_buffers 4 256k;
fastcgi_busy_buffers_size 256k;
fastcgi_temp_file_write_size 256k;
}
~/.ssh/config:
Host *
ConnectTimeout 2
TCPKeepAlive yes
Port 22
Identityfile ~/.ssh/server1000
ControlMaster no
ControlPath ~/.ssh/master-%r@%h:%p
Host server1000
Hostname 192.168.2.2
Creating connection to remote server:
# ssh -o "ServerAliveInterval 15" -o "ServerAliveCountMax 1" -MN user@server1000
# ls -la ~/.ssh/
srw------- 1 storage storage 0 Mar 26 15:46 master-root@192.168.2.2:22
# ssh server1000 whoami
root
and this simple script for which was run from browser and i'm getting same behaviour in Debian
and FreeBSD
<?php
for ($i = 0; $i < 100; ++$i) {
echo exec("ssh root@192.168.2.2 ls -la");
}
?>
[26-Mar-2013 15:57:00] NOTICE: configuration file /etc/php5/fpm/php-fpm.conf test is successful
[26-Mar-2013 15:57:00] NOTICE: fpm is running, pid 20053
[26-Mar-2013 15:57:00] NOTICE: ready to handle connections
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20054 exited with code 0 after 92.745193 seconds
from start
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20324 started
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20324 exited with code 0 after 0.005245 seconds
from start
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20325 started
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20325 exited with code 0 after 0.005121 seconds
from start
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20326 started
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20326 exited with code 0 after 0.005202 seconds
from start
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20327 started
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20327 exited with code 0 after 0.005543 seconds
from start
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20328 started
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20328 exited with code 0 after 0.004935 seconds
from start
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20329 started
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20329 exited with code 0 after 0.005255 seconds
from start
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20330 started
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20330 exited with code 0 after 0.005239 seconds
from start
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20331 started
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20331 exited with code 0 after 0.005134 seconds
from start
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20332 started
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20332 exited with code 0 after 0.005162 seconds
from start
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20333 started
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20333 exited with code 0 after 0.004950 seconds
from start
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20334 started
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20334 exited with code 0 after 0.005066 seconds
from start
[26-Mar-2013 15:58:33] NOTICE: [pool storage] child 20335 started
............
............
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=61558
--
Edit this bug report at https://bugs.php.net/bug.php?id=61558&edit=1