Edit report at https://bugs.php.net/bug.php?id=71135&edit=1
ID: 71135
Comment by: sroussey at gmail dot com
Reported by: iquito at gmx dot net
Summary: Random memory corruption with strings
Status: Closed
Type: Bug
Package: opcache
Operating System: Debian Jessie
PHP Version: 7.0.0
Assigned To: laruence
Block user comment: N
Private report: N
New Comment:
jjones: I wrote to nick at noodles up above about the memcache issue. Not sure why the issue is only
in 7.1 vs 7.0. We found high corruption rates in 5.6.x where x>9 as well. Never figured them all
out, but fast_shutdown=0 helped a bit. The 7.0.x series has been the most stable for us.
That said, I really don't recommend rsync as a deployment strategy. Feel free to message me for
other ideas for a high load site.
Previous Comments:
------------------------------------------------------------------------
[2017-09-11 10:24:23] jjones at smugmug dot com
Re: coming up with a reproducible use case that triggers a crash. We're working on that, since
it would be the ideal way to find/fix the problem. But no luck so far. So that's a "work
in progress".
Re: Valgrind, I thought about that and wasn't sure how usable the system would be in terms of
performance with that extra overhead added to the "php-fpm" processes. But at this point
I think it's worth a shot.
Re: disparity in the crash dump outputs when the root cause it memory corruption. Yup! I have two
more core dumps so far and I haven't found anything that links them together yet aside from
impossible values for various address pointers. I'm going to dig into them more and keep
searching though.
------------------------------------------------------------------------
[2017-09-11 05:16:48] rasmus@php.net
So two very different spots in your PHP code. Kind of expected for memory corruption. The corruption
happened earlier and it is somewhat random when it actually triggers a crash. I don't suppose
you can reproduce it reliably? If you can Valgrind might be able to spot the initial corruption.
I use a this little valgrind script to do a memcheck:
#!/bin/bash
USE_ZEND_ALLOC=0 valgrind --tool=memcheck --leak-check=yes --suppressions=/home/rasmus/.suppressions
--track-origins=yes --num-callers=30 --show-reachable=yes "$@"
You can skip the suppressions, although it isn't a bad idea to run it once on a trivial
phpinfo() or hello world script and record your suppressions just to clean up the output a bit.
Then I just do: memcheck php script.php
Of course, it is unlikely to be that simple, but you can also run the entire php-fpm or httpd under
memcheck. It runs very slowly, so don't do this on a production box, of course.
------------------------------------------------------------------------
[2017-09-11 01:19:31] jjones at smugmug dot com
Thanks for the pointer to the gdbinit file. There's some nice tools in there...
Here are the results from "zbacktrace" from the two core dumps I have.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000b64983 in zend_inline_hash_func (len=31426464, str=0x7fcf65800001 <error: Cannot
access memory at address 0x7fcf65800001>) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_string.h:331
331 hash = ((hash << 5) + hash) + *str++;
(gdb) zbacktrace
[0x7fcf656146a0] Lib\Base->__autoload("SmugMug\Router\Img")
/var/www/www_inside/1504876151/include/setup.mgi:731
[0x7fcf65614640] spl_autoload_call("SmugMug\Router\Img") [internal function]
[0x7fff7073b0a0] ???
[0x7fcf656145a0] SmugMug\Router\Base->setup()
/var/www/www_inside/1504876151/include/classes/SmugMug/Router/Base.php:607
[0x7fcf656140e0] (main) /var/www/www_inside/1504876151/include/setup.mgi:100
[0x7fcf65614030] (main) /var/www/www_inside/1504876151/index.mg:8
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000bb6763 in ZEND_ECHO_SPEC_CONST_HANDLER () at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:2667
2667 if (ZSTR_LEN(str) != 0) {
(gdb) zbacktrace
[0x7f8f2f415430] Lib\Spit->(main)
/var/www/www_inside/1504876151/include/legacy-pages/test/react-library/react-tools-browserify-bundle.js:1
[0x7f8f2f4153a0]
Lib\Spit->out("/include/legacy-pages/test/react-library/react-tools-browserify-bundle.js",
reference) /var/www/www_inside/1504876151/include/classes/Lib/Spit.php:39
[0x7f8f2f415300]
Lib\Spit->swallow("/include/legacy-pages/test/react-library/react-tools-browserify-bundle.js",
array(0)[0x7f8f2f415360]) /var/www/www_inside/1504876151/include/classes/Lib/Spit.php:57
[0x7f8f2f415270]
SmugMug\Spit->swallow("/include/legacy-pages/test/react-library/react-tools-browserify-bundle.js")
/var/www/www_inside/1504876151/include/classes/SmugMug/Spit.php:61
[0x7f8f2f415150] SmugMug\UI\UIExample->init(object[0x7f8f2f4151a0], array(1)[0x7f8f2f4151b0])
/var/www/www_inside/1504876151/include/classes/SmugMug/UI/UIExample.php:54
[0x7f8f2f4150e0] (main)
/var/www/www_inside/1504876151/include/legacy-pages/test/react-library/index.mg:94
[0x7f8f2f415030] (main) /var/www/www_inside/1504876151/index.mg:21
------------------------------------------------------------------------
[2017-09-10 13:25:33] rasmus@php.net
Do you get a useful zbacktrace from these cores? (See https://github.com/php/php-src/blob/PHP-7.1/.gdbinit)
------------------------------------------------------------------------
[2017-09-10 09:18:17] jjones at smugmug dot com
As for the values of the opcache restart stats from the crashes processes, I think this is what
you're asking for.
#0 0x0000000000b64983 in zend_inline_hash_func (len=31426464,
str=0x7fcf65800001 <error: Cannot access memory at address 0x7fcf65800001>)
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_string.h:331
331 hash = ((hash << 5) + hash) + *str++;
(gdb) print *accel_shared_globals
$1 = {hits = 573580, misses = 94, blacklist_misses = 0, oom_restarts = 0, hash_restarts = 0,
manual_restarts = 0
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000bb6763 in ZEND_ECHO_SPEC_CONST_HANDLER ()
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:2667
2667 if (ZSTR_LEN(str) != 0) {
(gdb) print *accel_shared_globals
$1 = {hits = 19012587, misses = 53, blacklist_misses = 0, oom_restarts = 0, hash_restarts = 0,
manual_restarts = 0
Meaning, they were all zero at the time, in both core dumps.
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=71135
--
Edit this bug report at https://bugs.php.net/bug.php?id=71135&edit=1