Edit report at https://bugs.php.net/bug.php?id=71135&edit=1
ID: 71135
Comment by: nick at noodles dot net dot nz
Reported by: iquito at gmx dot net
Summary: Random memory corruption with strings
Status: Closed
Type: Bug
Package: opcache
Operating System: Debian Jessie
PHP Version: 7.0.0
Assigned To: laruence
Block user comment: N
Private report: N
New Comment:
I'm still seeing problems as well. So far I've disabled huge code pages and changed
opcache to not check timestamps (code deploys now fire an opcache_reset() on each webserver).
Is it possible to disable interned strings?
Also, should we be adding comments to this bug or creating a new one (being that this is closed)?
Previous Comments:
------------------------------------------------------------------------
[2017-09-12 17:21:51] mike@php.net
After many more tests, it seems it rather depends on the moon phase, because I cannot reproduce
anymore.
Looking into the write of uninitialized bytes next.
------------------------------------------------------------------------
[2017-09-12 13:13:55] mike@php.net
Some valgrind logs:
https://gist.github.com/m6w6/67f71247f5325497a59d4686ba4845ac
------------------------------------------------------------------------
[2017-09-12 12:19:15] mike@php.net
The following patch has been added/updated:
Patch Name: interned_strings_top
Revision: 1505218752
URL: https://bugs.php.net/patch-display.php?bug=71135&patch=interned_strings_top&revision=1505218752
------------------------------------------------------------------------
[2017-09-12 12:18:03] mike@php.net
Several ingredients are needed to reproduce:
* opcache.file_cache enabled
* opcache.interned_strings_buffer set low (eg. 1)
* a reasonable large codebase to fill up the interned strings buffer
* quite a bit of luck that the remaining free buffer size is low enough to trigger
I could reproducibly, but not quite reliably produce a crash or zend_mm_panic with that recipe with
eg. laravel or phan.
I've way too less knowledge of opcache to argue yet what's going on, but I'll upload
a patch which /seemed/ to fix the issue for me. Maybe someone with more insight can comment whether
I'm totally off.
------------------------------------------------------------------------
[2017-09-12 04:28:21] jjones at smugmug dot com
I failed mention it in my original posting, but we are running with "fast_shutdown=0"
already. That suggestion popped up in a debugging posting I read early on in the process.
We found a way to reproduce the SegFault's we're seeing on our PHP 7.1.9 servers today.
The servers running a normal configuration still see "php-fpm" crashes from time to time.
They are infrequent however so iterative debugging has been slow and tedious.
But we found that by reducing the "opcache.interned_strings_buffer" setting the time
between crashes was greatly reduced. Checking the resulting core dumps in "gdb" with,
set $total = (accel_shared_globals->interned_strings_end -
accel_shared_globals->interned_strings_start) + 1
set $used = accel_shared_globals->interned_strings_top -
accel_shared_globals->interned_strings_start
set $left = (accel_shared_globals->interned_strings_end -
accel_shared_globals->interned_strings_top) + 1
set $prev = accel_shared_globals->interned_strings_end -
accel_shared_globals->interned_strings_saved_top
printf "Interned Strings Total Memory: %10d\n", $total
printf "Interned Strings Used Memory: %10d\n", $used
printf "Interned Strings Free Memory: %10d\n", $left
printf "Interned Strings Previous Free: %10d\n", $prev
shows that the buffer reserved for interned strings is exhausted when each of them crashed. Going
back to the core dumps from the "normal" configurations, we found that they showed similar
stats with respect to the available interned string buffer space.
In other words, the one things the core dumps seem to have in common is that they have almost no
free space in their interned string buffers when they attempt to derefernce an invalid address
pointer. The backtraces differ, the code which triggers the SegFault, and the types of structures
with invalid pointers varied. But in the six core dumps we captured today, three from a normal
configuration and three from a system with very limited interned string buffers, when they crashed
there was very little space available in the buffer space.
- - - Core file: php-fpm719-170911:1317.dump
[New LWP 4163]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `php-fpm: pool www '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000b3bc76 in zend_string_hash_val (s=0x7f97754e22b0) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_string.h:85
85 if (!ZSTR_H(s)) {
Interned Strings Total Memory: 4194305
Interned Strings Used Memory: 4194280
Interned Strings Free Memory: 25
Interned Strings Previous Free: 3856224
- - - Core file: php-fpm719-170911:1332.dump
[New LWP 15871]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `php-fpm: pool www '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000c72bc5 in ZEND_ISSET_ISEMPTY_DIM_OBJ_SPEC_CV_CV_HANDLER () at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:47096
47096 if (ZEND_HANDLE_NUMERIC(str, hval)) {
Interned Strings Total Memory: 4194305
Interned Strings Used Memory: 4194296
Interned Strings Free Memory: 9
Interned Strings Previous Free: 3856224
- - - Core file: php-fpm719-170911:1336.dump
[New LWP 13330]
[Thread debugging using libthread_db enabled]
xcUsing host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `php-fpm: pool www '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000c72bc5 in ZEND_ISSET_ISEMPTY_DIM_OBJ_SPEC_CV_CV_HANDLER () at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:47096
47096 if (ZEND_HANDLE_NUMERIC(str, hval)) {
Interned Strings Total Memory: 4194305
Interned Strings Used Memory: 4194296
Interned Strings Free Memory: 9
Interned Strings Previous Free: 3856224
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=71135
--
Edit this bug report at https://bugs.php.net/bug.php?id=71135&edit=1