Bug #66694 [Com]: php-fpm is crashing when using eventport

From: Date: Mon, 06 Mar 2017 19:09:59 +0000
Subject: Bug #66694 [Com]: php-fpm is crashing when using eventport
References: 1  Groups: php.bugs 
Request: Send a blank email to php-bugs+get-207711@lists.php.net to get a copy of this message
Edit report at https://bugs.php.net/bug.php?id=66694&edit=1

 ID:                 66694
 Comment by:         stadtkind2 at gmx dot de
 Reported by:        eugene at zhegan dot in
 Summary:            php-fpm is crashing when using eventport
 Status:             Open
 Type:               Bug
 Package:            FPM related
 Operating System:   Solaris 11.1
 PHP Version:        5.4.25
 Block user comment: N
 Private report:     N

 New Comment:

The 03-event_port.patch patch at https://github.com/OpenIndiana/oi-userland/tree/oi/hipster/components/web/php/php-7_0/patches
works better, no core dumps anymore.


Previous Comments:
------------------------------------------------------------------------
[2017-03-02 11:32:48] stadtkind2 at gmx dot de

I have the same problem with PHP 7.1.2 on Solaris 11.3/SPARC. The applied patch did not fix it,
I'm still getting coredumps:

# pstack /var/cores/core.php-fpm.0.15837
core '/var/cores/core.php-fpm.0.15837' of 15837:        /opt/php7/sbin/php-fpm
 0000000100caa288 fpm_event_fire (0, 0, 1, 20000, 0, 101076720) + 8
 0000000100caa0ac fpm_event_loop (0, 0, ffffffff7ffff550, 100de4880, 10059c5c8, fffffffffff0bdc0) +
2fc
 0000000100ca0024 fpm_run (ffffffff7ffffa24, bd350, 0, bd000, 6, ffffffff7f782a40) + 64
 0000000100cb22cc main (7b9400, ffffffffff7d8674, 827800, 0, 100dc3fb0, ffffffffff846a18) + eec
 000000010060dee8 _start (0, 0, 0, 0, 0, 100dc3fb0) + 108

------------------------------------------------------------------------
[2016-11-09 19:47:35] christian at kuehnke dot de

I experienced this problem (php-fpm SIGSEGVs after sending it SIGQUIT immediately after startup)
with php 7.0.12 and 5.6.27 on Solaris 11.3.

I attached a one-liner patch for this problem. It is still not well tested, will update here
later...

------------------------------------------------------------------------
[2014-05-02 13:56:22] d dot v dot taylor at leedsmet dot ac dot uk

I've hit this too on 5.5.11, but with a Bus Error rather than a SEGV (which will be a SPARC vs
x86 thing). It particularly happens if I kill -QUIT FPM before it's handled *any* requests (and
so hasn't received any events). This is the backtrace (using mdb):

fpm_event_fire+0x1c(2e70687035002e, ffffffff, 1, ffffffff7fffd470, ffffffff7fffd460,
ffffffffffffffff)
fpm_event_port_wait+0x1ec(1014a3c50, 3e8, 2, 100efe858, 1768, 0)
fpm_event_loop+0x3f0(0, 0, 0, 0, 0, 0)
fpm_run+0xa8(ffffffff7ffff928, 0, a, 100effc80, ffffffff7df02a40, ffffffff7df02a40)
main+0x12bc(1, ffffffffffffffff, ffffffff7ffffaa8, ffffffff7df001c0, 2880, 2800)
_start+0x12c(0, 0, 0, 0, 0, 0)



This is what I think is going on in my case (maybe yours too?) - long story short it's a memory
reuse problem:

fpm_event_port_init() malloc()s a chunk of memory to store events retrieved from the port, and
doesn't initialise that memory at all. In my case it happened to have previously stored a
pointer to a string used in processing php-fpm.conf.

fpm_event_port_wait() uses port_getn() to look for events, with a parameter nget  - the intended
number of events - set to 1. In normal use that nget parameter is updated to the number of events
actually retrieved, but if port_getn is interrupted by a signal and no events have been sent to the
port, nget is left at 1 (see http://marc.info/?l=opensolaris-networking-discuss&m=125441701003618)

The code below the port_getn() call sees nget > 0, assumes an event has been returned, and tries
to process it. Because the memory assigned to store events hasn't been zeroed out, it gets
through the brief sanity check and passes a junk pointer to fpm_event_fire - which then reads the
random string (in my case ".php5\0.") as itself being a pointer (0x2e70687035002e), tries
to dereference it and falls in a heap.

I'm not sure if it's a safe assumption that if port_getn() was interrupted by a signal and
we're only asking for one event, then there *definitely* won't have been any events
retrieved - I suspect not. I also don't know if just zeroing out the memory in
fpm_event_port_init() is enough, or if that only works because I'm only hitting it immediately
after startup.

------------------------------------------------------------------------
[2014-02-11 08:57:37] eugene at zhegan dot in

Description:
------------
php-fpm is crashing when 'events.mechanism' is set to 'port'. It doesn't
crash on first request, it can process some, however, the crash is imminent and my installation
dumps cores like 1 per minute (this server is used by ~50 developers).

When set to 'events.mechanism = /dev/poll' php-fpm stop to crash.

Backtrace (they all do look similar):

# /usr/local/solarisstudio12.3/bin/dbx /usr/local/php-5.4.25/sbin/php-fpm
core_hyperion_php-fpm_0_%c_5046 
For information about new features see `help changes'
To remove this message, put `dbxenv suppress_startup_message 7.9' in your .dbxrc
Reading php-fpm
core file header read successfully
Reading ld.so.1
Reading preloadable_libiconv.so
Reading libcrypt.so.1
Reading libresolv.so.2
Reading librt.so.1
Reading libmysqlclient.so.18.0.0
Reading libsybdb.so.5.0.0
Reading libnsl.so.1
Reading libsocket.so.1
Reading libpthread.so.1
Reading libldap.so.5
Reading liblber-2.4.so.2.8.3
Reading libpng15.so.15.13.0
Reading libz.so.1
Reading libm.so.2
Reading libjpeg.so.62.0.0
Reading libcrypto.so.1.0.0
Reading libssl.so.1.0.0
Reading libxml2.so.2
Reading libfreetype.so.6
Reading libc.so.1
Reading libgcc_s.so.1
Reading libdl.so.1
Reading libnspr4.so
Reading libmd.so.1
Reading libmp.so.2
Reading libsasl.so.1
Reading libplc4.so
Reading libnss3.so
Reading libnssutil3.so
Reading libplds4.so
Reading libssl3.so
Reading libthread.so.1
Reading libbz2.so.1
Reading libcryptoutil.so.1
Reading ru_RU.UTF-8.so.3
Reading methods_unicode.so.3
Reading xdebug.so
Reading bz2.so
Reading curl.so
Reading libcurl.so.4.2.0
Reading imap.so
Reading libpam.so.1
Reading memcache.so
Reading mcrypt.so
Reading libmcrypt.so.4.4.8
Reading libltdl.so.7.3.0
Reading oci8.so
Reading libclntsh.so.11.1
dbx: internal warning: "(null)"::srcname(InfoOrigin_CMDLINE): value "ntcontab.c"
being set to "nt.c"
Reading libnnz11.so
Reading libkstat.so.1
Reading libgen.so.1
Reading libsched.so.1
Reading libaio.so.1
Reading libpool.so.1
Reading openssl.so
Reading pinba.so
Reading redis.so
Reading xmlrpc.so
Reading xsl.so
Reading libexslt.so.0
Reading libxslt.so.1
Reading nss_files.so.1
Reading nss_ldap.so.1
Reading libsldap.so.1
Reading libscf.so.1
t@1 (l@1) program terminated by signal SEGV (no mapping at the fault address)
0x00000000008f75d7: fpm_event_fire+0x0009:      movq     0x0000000000000028(%rdi),%rax
(dbx) where                                                                  
current thread: t@1
=>[1] fpm_event_fire(0x75723d63642c61, 0x26, 0x0, 0x1, 0x4, 0xffff80ffbfffd0b0), at 0x8f75d7 
  [2] fpm_event_port_wait(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x13b6, 0x7d141, 0xe18ed, 0x1, 0x0,
0xffff80ffbffffb3c, 0xffff80ffbffffc18, 0x1, 0xd55d0c, 0xffff80ffbfffd160, 0x8f2925, 0xe23c20, 0x0),
at 0x902998 
  [3] fpm_event_loop(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0x8f7927 
  [4] fpm_run(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0x8f2925 
  [5] main(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0x8fa704 
(dbx)


Another one:

# /usr/local/solarisstudio12.3/bin/dbx /usr/local/php-5.4.25/sbin/php-fpm
core_hyperion_php-fpm_0_%c_4964
For information about new features see `help changes'
To remove this message, put `dbxenv suppress_startup_message 7.9' in your .dbxrc
Reading php-fpm
core file header read successfully
Reading ld.so.1
Reading preloadable_libiconv.so
Reading libcrypt.so.1
Reading libresolv.so.2
Reading librt.so.1
Reading libmysqlclient.so.18.0.0
Reading libsybdb.so.5.0.0
Reading libnsl.so.1
Reading libsocket.so.1
Reading libpthread.so.1
Reading libldap.so.5
Reading liblber-2.4.so.2.8.3
Reading libpng15.so.15.13.0
Reading libz.so.1
Reading libm.so.2
Reading libjpeg.so.62.0.0
Reading libcrypto.so.1.0.0
Reading libssl.so.1.0.0
Reading libxml2.so.2
Reading libfreetype.so.6
Reading libc.so.1
Reading libgcc_s.so.1
Reading libdl.so.1
Reading libnspr4.so
Reading libmd.so.1
Reading libmp.so.2
Reading libsasl.so.1
Reading libplc4.so
Reading libnss3.so
Reading libnssutil3.so
Reading libplds4.so
Reading libssl3.so
Reading libthread.so.1
Reading libbz2.so.1
Reading libcryptoutil.so.1
Reading ru_RU.UTF-8.so.3
Reading methods_unicode.so.3
Reading xdebug.so
Reading bz2.so
Reading curl.so
Reading libcurl.so.4.2.0
Reading imap.so
Reading libpam.so.1
Reading memcache.so
Reading mcrypt.so
Reading libmcrypt.so.4.4.8
Reading libltdl.so.7.3.0
Reading oci8.so
Reading libclntsh.so.11.1
dbx: internal warning: "(null)"::srcname(InfoOrigin_CMDLINE): value "ntcontab.c"
being set to "nt.c"
Reading libnnz11.so
Reading libkstat.so.1
Reading libgen.so.1
Reading libsched.so.1
Reading libaio.so.1
Reading libpool.so.1
Reading openssl.so
Reading pinba.so
Reading redis.so
Reading xmlrpc.so
Reading xsl.so
Reading libexslt.so.0
Reading libxslt.so.1
Reading nss_files.so.1
Reading nss_ldap.so.1
Reading libsldap.so.1
Reading libscf.so.1
t@1 (l@1) program terminated by signal SEGV (no mapping at the fault address)
0x00000000008f75d7: fpm_event_fire+0x0009:      movq     0x0000000000000028(%rdi),%rax
(dbx) where                                                                  
current thread: t@1
=>[1] fpm_event_fire(0x75723d63642c61, 0x26, 0x0, 0x1, 0x4, 0xffff80ffbfffd0b0), at 0x8f75d7 
  [2] fpm_event_port_wait(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x1364, 0x7d0da, 0xb7ce3, 0x1, 0x0,
0xffff80ffbffffb3c, 0xffff80ffbffffc18, 0x1, 0xd55d0c, 0xffff80ffbfffd160, 0x8f2925, 0xe23c20, 0x0),
at 0x902998 
  [3] fpm_event_loop(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0x8f7927 
  [4] fpm_run(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0x8f2925 
  [5] main(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0x8fa704 
(dbx)



------------------------------------------------------------------------



--
Edit this bug report at https://bugs.php.net/bug.php?id=66694&edit=1


Thread (6 messages)

« previous php.bugs (#207711) next »