Req #76225 [Com]: ini setting to not empty opcache when php-fpm is reloaded

Edit report at https://bugs.php.net/bug.php?id=76225&edit=1 ID: 76225 Comment by: post at minhost dot no Reported by: post at minhost dot no Summary: ini setting to not empty opcache when php-fpm is reloaded Status: Open Type: Feature/Change Request Package: opcache Operating System: CentOS 7.4 PHP Version: 7.1.16 Block user comment: N Private report: N New Comment: On the test server I have managed to crash two test sites now Some observations I have done now, is that each time I get crash on the test server, it is no more free memory in opcahe, like this (the test server does not have a lot of memory): Used memory 134214488 Free memory 3240 However on the production servers, I have allocated 32 GB to opcache, and I have never seen it use more then 9 GB in opcache. However I am not sure how opcache user the memory limit I set, because the 32 GB is not visible when looking at used memory in shell: [root@production-server ~]# free -m total used free shared buff/cache available Mem: 128658 14825 1961 6672 111870 106292 Swap: 955 954 1 Could it be that opcache is not able to allocate memory from the already used buffer/cache above? Or is the 32 GB memory limit in opcache already accounted for in the current buffer/cache? Anyway, I am still not able to get any dump files in /tmp on the test server. Previous Comments: ------------------------------------------------------------------------ [2018-04-20 16:02:40] post at minhost dot no By the way, when I manually double check by looking into the file: /proc/sys/kernel/core_pattern It correctly has this content: /tmp/core-%e.%p And looking into: /proc/sys/kernel/core_uses_pid It correctly has this content: 0 ------------------------------------------------------------------------ [2018-04-20 15:57:37] post at minhost dot no I was finnaly able to get a crash of the copy of a site on a test server. However there was no dump files in /tmp - I really don't know what I did wrong? Here is what I did to activate bactrace: # First I changed all php-fpm.conf config files to include this: rlimit_core = unlimited # Then I compiled PHP-FPM with --enable-debug - and I confirmed a PHP info page says this: "Debug Build yes" # Then I installed GDB with yum: yum install gdb # Then I enabled core dumps in linux like this: su - echo '/tmp/core-%e.%p' > /proc/sys/kernel/core_pattern echo 0 > /proc/sys/kernel/core_uses_pid ulimit -c unlimited # Then i logged out of shell. Then a few days passed without any crash on the test site on the test server. Then today I was able to reproduce a crash on the test server, and PHP-FPM for the site crashed with a 503 error. In apache error log for the test site I got this: [Fri Apr 20 17:30:02.395339 2018] [proxy_fcgi:error] [pid 19834:tid 140118967469824] [client 176.74.214.18:6917] AH01067: Failed to read FastCGI header [Fri Apr 20 17:30:02.395398 2018] [proxy_fcgi:error] [pid 19834:tid 140118967469824] (104)Connection reset by peer: [client 176.74.214.18:6917] AH01075: Error dispatching request to : [Fri Apr 20 17:30:45.711350 2018] [proxy_fcgi:error] [pid 19834:tid 140119471032064] [client 176.74.214.18:6933] AH01067: Failed to read FastCGI header, referer: https://www.fjell-bfk4k.41.no/update.php?op=info [Fri Apr 20 17:30:45.711443 2018] [proxy_fcgi:error] [pid 19834:tid 140119471032064] (104)Connection reset by peer: [client 176.74.214.18:6933] AH01075: Error dispatching request to : , referer: https://www.fjell-bfk4k.41.no/update.php?op=info # Then I went into /tmp to look for core dumps, but ther are not any core dumps in /tmp! By the way I can run gdb just fine like this [root@dns2 ~]# gdb GNU gdb (GDB) Red Hat Enterprise Linux 7.6.1-100.el7_4.1 Copyright (C) 2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-redhat-linux-gnu". For bug reporting instructions, please see: <http://www.gnu.org/software/gdb/bugs/>. (gdb) But of course I just get: (gdb) bt No stack. (gdb) help I would like to run: gdb /usr/local/php71/sbin/php-fpm71 /tmp/file-name-to-dump-file.123123 But of course I can't, because there is no dump files in /tmp What am I doing wrong? I really need som help here. ------------------------------------------------------------------------ [2018-04-17 11:24:52] nikic@php.net It is not necessary to build PHP with --enable-debug (which apart from enabling debug symbols also disables optimization and enables debug assertions). The documentation and ./configure --help output are quite misleading in that regard and should be improved. For a basic backtrace without file+line information, just using a normal PHP build is fine. To also have file+line information it's possible to pass CFLAGS="-g" to configure, in which case PHP will be built with debug symbols (and optimization). This should not impact performance, but increases the size of the binary. ------------------------------------------------------------------------ [2018-04-17 10:23:16] post at minhost dot no When compiling PHP with --enable-debug, PHP runs to slow for me to do this on production servers. As I said, the crash only happen sporadic, and it can be anything from days to weeks before it happen again. So I can't do the backtrace on my production servers. Instead I will try to setup a copy of the sites on a test VPS, and run backtrace on that test server instead. I can only hope I will be able to trigger the crashes when it is only a copy of the live sites. I will then try to reproduce the crashes of both InfititeWP in this bug https://bugs.php.net/bug.php?id=76205 and on Drupal sites when running update.php (described in the bug report I am commenting on now). Please note that this bug report was a feature request. But I guess it does not matter if I use the comment field here. This could take time ... ------------------------------------------------------------------------ [2018-04-16 13:25:52] nikic@php.net Ah yes, you will have to install gdb first (using "sudo apt-get install gdb" or whatever the equivalent for your operating system is). Alternatively, if you do not want to install additional software on a production server, it is also in principle possible to copy the core dump to a different machine that has gdb installed and analyze it there. However, this requires that both the php-fpm binary and shared objects are the same on the other machine. This tends to be more finicky, so if you can directly install gdb, I'd recommend going that way. ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=76225 -- Edit this bug report at https://bugs.php.net/bug.php?id=76225&edit=1