Edit report at https://bugs.php.net/bug.php?id=71135&edit=1
ID: 71135
Comment by: jjones at smugmug dot com
Reported by: iquito at gmx dot net
Summary: Random memory corruption with strings
Status: Closed
Type: Bug
Package: opcache
Operating System: Debian Jessie
PHP Version: 7.0.0
Assigned To: laruence
Block user comment: N
Private report: N
New Comment:
I failed mention it in my original posting, but we are running with "fast_shutdown=0"
already. That suggestion popped up in a debugging posting I read early on in the process.
We found a way to reproduce the SegFault's we're seeing on our PHP 7.1.9 servers today.
The servers running a normal configuration still see "php-fpm" crashes from time to time.
They are infrequent however so iterative debugging has been slow and tedious.
But we found that by reducing the "opcache.interned_strings_buffer" setting the time
between crashes was greatly reduced. Checking the resulting core dumps in "gdb" with,
set $total = (accel_shared_globals->interned_strings_end -
accel_shared_globals->interned_strings_start) + 1
set $used = accel_shared_globals->interned_strings_top -
accel_shared_globals->interned_strings_start
set $left = (accel_shared_globals->interned_strings_end -
accel_shared_globals->interned_strings_top) + 1
set $prev = accel_shared_globals->interned_strings_end -
accel_shared_globals->interned_strings_saved_top
printf "Interned Strings Total Memory: %10d\n", $total
printf "Interned Strings Used Memory: %10d\n", $used
printf "Interned Strings Free Memory: %10d\n", $left
printf "Interned Strings Previous Free: %10d\n", $prev
shows that the buffer reserved for interned strings is exhausted when each of them crashed. Going
back to the core dumps from the "normal" configurations, we found that they showed similar
stats with respect to the available interned string buffer space.
In other words, the one things the core dumps seem to have in common is that they have almost no
free space in their interned string buffers when they attempt to derefernce an invalid address
pointer. The backtraces differ, the code which triggers the SegFault, and the types of structures
with invalid pointers varied. But in the six core dumps we captured today, three from a normal
configuration and three from a system with very limited interned string buffers, when they crashed
there was very little space available in the buffer space.
- - - Core file: php-fpm719-170911:1317.dump
[New LWP 4163]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `php-fpm: pool www '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000b3bc76 in zend_string_hash_val (s=0x7f97754e22b0) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_string.h:85
85 if (!ZSTR_H(s)) {
Interned Strings Total Memory: 4194305
Interned Strings Used Memory: 4194280
Interned Strings Free Memory: 25
Interned Strings Previous Free: 3856224
- - - Core file: php-fpm719-170911:1332.dump
[New LWP 15871]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `php-fpm: pool www '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000c72bc5 in ZEND_ISSET_ISEMPTY_DIM_OBJ_SPEC_CV_CV_HANDLER () at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:47096
47096 if (ZEND_HANDLE_NUMERIC(str, hval)) {
Interned Strings Total Memory: 4194305
Interned Strings Used Memory: 4194296
Interned Strings Free Memory: 9
Interned Strings Previous Free: 3856224
- - - Core file: php-fpm719-170911:1336.dump
[New LWP 13330]
[Thread debugging using libthread_db enabled]
xcUsing host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `php-fpm: pool www '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000c72bc5 in ZEND_ISSET_ISEMPTY_DIM_OBJ_SPEC_CV_CV_HANDLER () at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:47096
47096 if (ZEND_HANDLE_NUMERIC(str, hval)) {
Interned Strings Total Memory: 4194305
Interned Strings Used Memory: 4194296
Interned Strings Free Memory: 9
Interned Strings Previous Free: 3856224
Previous Comments:
------------------------------------------------------------------------
[2017-09-11 22:55:39] sroussey at gmail dot com
jjones: I wrote to nick at noodles up above about the memcache issue. Not sure why the issue is only
in 7.1 vs 7.0. We found high corruption rates in 5.6.x where x>9 as well. Never figured them all
out, but fast_shutdown=0 helped a bit. The 7.0.x series has been the most stable for us.
That said, I really don't recommend rsync as a deployment strategy. Feel free to message me for
other ideas for a high load site.
------------------------------------------------------------------------
[2017-09-11 10:24:23] jjones at smugmug dot com
Re: coming up with a reproducible use case that triggers a crash. We're working on that, since
it would be the ideal way to find/fix the problem. But no luck so far. So that's a "work
in progress".
Re: Valgrind, I thought about that and wasn't sure how usable the system would be in terms of
performance with that extra overhead added to the "php-fpm" processes. But at this point
I think it's worth a shot.
Re: disparity in the crash dump outputs when the root cause it memory corruption. Yup! I have two
more core dumps so far and I haven't found anything that links them together yet aside from
impossible values for various address pointers. I'm going to dig into them more and keep
searching though.
------------------------------------------------------------------------
[2017-09-11 05:16:48] rasmus@php.net
So two very different spots in your PHP code. Kind of expected for memory corruption. The corruption
happened earlier and it is somewhat random when it actually triggers a crash. I don't suppose
you can reproduce it reliably? If you can Valgrind might be able to spot the initial corruption.
I use a this little valgrind script to do a memcheck:
#!/bin/bash
USE_ZEND_ALLOC=0 valgrind --tool=memcheck --leak-check=yes --suppressions=/home/rasmus/.suppressions
--track-origins=yes --num-callers=30 --show-reachable=yes "$@"
You can skip the suppressions, although it isn't a bad idea to run it once on a trivial
phpinfo() or hello world script and record your suppressions just to clean up the output a bit.
Then I just do: memcheck php script.php
Of course, it is unlikely to be that simple, but you can also run the entire php-fpm or httpd under
memcheck. It runs very slowly, so don't do this on a production box, of course.
------------------------------------------------------------------------
[2017-09-11 01:19:31] jjones at smugmug dot com
Thanks for the pointer to the gdbinit file. There's some nice tools in there...
Here are the results from "zbacktrace" from the two core dumps I have.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000b64983 in zend_inline_hash_func (len=31426464, str=0x7fcf65800001 <error: Cannot
access memory at address 0x7fcf65800001>) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_string.h:331
331 hash = ((hash << 5) + hash) + *str++;
(gdb) zbacktrace
[0x7fcf656146a0] Lib\Base->__autoload("SmugMug\Router\Img")
/var/www/www_inside/1504876151/include/setup.mgi:731
[0x7fcf65614640] spl_autoload_call("SmugMug\Router\Img") [internal function]
[0x7fff7073b0a0] ???
[0x7fcf656145a0] SmugMug\Router\Base->setup()
/var/www/www_inside/1504876151/include/classes/SmugMug/Router/Base.php:607
[0x7fcf656140e0] (main) /var/www/www_inside/1504876151/include/setup.mgi:100
[0x7fcf65614030] (main) /var/www/www_inside/1504876151/index.mg:8
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000bb6763 in ZEND_ECHO_SPEC_CONST_HANDLER () at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:2667
2667 if (ZSTR_LEN(str) != 0) {
(gdb) zbacktrace
[0x7f8f2f415430] Lib\Spit->(main)
/var/www/www_inside/1504876151/include/legacy-pages/test/react-library/react-tools-browserify-bundle.js:1
[0x7f8f2f4153a0]
Lib\Spit->out("/include/legacy-pages/test/react-library/react-tools-browserify-bundle.js",
reference) /var/www/www_inside/1504876151/include/classes/Lib/Spit.php:39
[0x7f8f2f415300]
Lib\Spit->swallow("/include/legacy-pages/test/react-library/react-tools-browserify-bundle.js",
array(0)[0x7f8f2f415360]) /var/www/www_inside/1504876151/include/classes/Lib/Spit.php:57
[0x7f8f2f415270]
SmugMug\Spit->swallow("/include/legacy-pages/test/react-library/react-tools-browserify-bundle.js")
/var/www/www_inside/1504876151/include/classes/SmugMug/Spit.php:61
[0x7f8f2f415150] SmugMug\UI\UIExample->init(object[0x7f8f2f4151a0], array(1)[0x7f8f2f4151b0])
/var/www/www_inside/1504876151/include/classes/SmugMug/UI/UIExample.php:54
[0x7f8f2f4150e0] (main)
/var/www/www_inside/1504876151/include/legacy-pages/test/react-library/index.mg:94
[0x7f8f2f415030] (main) /var/www/www_inside/1504876151/index.mg:21
------------------------------------------------------------------------
[2017-09-10 13:25:33] rasmus@php.net
Do you get a useful zbacktrace from these cores? (See https://github.com/php/php-src/blob/PHP-7.1/.gdbinit)
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=71135
--
Edit this bug report at https://bugs.php.net/bug.php?id=71135&edit=1