Edit report at https://bugs.php.net/bug.php?id=71135&edit=1
ID: 71135
Updated by: rasmus@php.net
Reported by: iquito at gmx dot net
Summary: Random memory corruption with strings
Status: Closed
Type: Bug
Package: opcache
Operating System: Debian Jessie
PHP Version: 7.0.0
Assigned To: laruence
Block user comment: N
Private report: N
New Comment:
Do you get a useful zbacktrace from these cores? (See https://github.com/php/php-src/blob/PHP-7.1/.gdbinit)
Previous Comments:
------------------------------------------------------------------------
[2017-09-10 09:18:17] jjones at smugmug dot com
As for the values of the opcache restart stats from the crashes processes, I think this is what
you're asking for.
#0 0x0000000000b64983 in zend_inline_hash_func (len=31426464,
str=0x7fcf65800001 <error: Cannot access memory at address 0x7fcf65800001>)
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_string.h:331
331 hash = ((hash << 5) + hash) + *str++;
(gdb) print *accel_shared_globals
$1 = {hits = 573580, misses = 94, blacklist_misses = 0, oom_restarts = 0, hash_restarts = 0,
manual_restarts = 0
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000bb6763 in ZEND_ECHO_SPEC_CONST_HANDLER ()
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:2667
2667 if (ZSTR_LEN(str) != 0) {
(gdb) print *accel_shared_globals
$1 = {hits = 19012587, misses = 53, blacklist_misses = 0, oom_restarts = 0, hash_restarts = 0,
manual_restarts = 0
Meaning, they were all zero at the time, in both core dumps.
------------------------------------------------------------------------
[2017-09-10 09:14:18] jjones at smugmug dot com
I captured another core dump which has different backtrace, but ultimately crashes with a SegFault
due to a string pointer is can't dereference.
#0 0x0000000000bb6763 in ZEND_ECHO_SPEC_CONST_HANDLER ()
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:2667
#1 0x0000000000ba9a6b in execute_ex (ex=0x7f8f2f415430)
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#2 0x0000000000c88ec8 in ZEND_INCLUDE_OR_EVAL_SPEC_TMPVAR_HANDLER ()
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:51699
#3 0x0000000000ba9a6b in execute_ex (ex=0x7f8f2f4153a0)
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#4 0x0000000000baec11 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:1076
#5 0x0000000000ba9a6b in execute_ex (ex=0x7f8f2f415300)
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#6 0x0000000000baec11 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:1076
#7 0x0000000000ba9a6b in execute_ex (ex=0x7f8f2f415270)
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#8 0x0000000000baec11 in ZEND_DO_FCALL_SPEC_RETVAL_USED_HANDLER ()
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:1076
#9 0x0000000000ba9a6b in execute_ex (ex=0x7f8f2f415150)
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#10 0x0000000000badfa4 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER ()
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:949
#11 0x0000000000ba9a6b in execute_ex (ex=0x7f8f2f4150e0)
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#12 0x0000000000c88ec8 in ZEND_INCLUDE_OR_EVAL_SPEC_TMPVAR_HANDLER ()
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:51699
#13 0x0000000000ba9a6b in execute_ex (ex=0x7f8f2f415030)
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#14 0x0000000000baa42b in zend_execute (op_array=0x7f8f2f4731c0, return_value=0x0)
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:474
#15 0x0000000000b0fcc1 in zend_execute_scripts (type=8, retval=0x0, file_count=3)
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend.c:1480
#16 0x0000000000a5b290 in php_execute_script (primary_file=0x7ffc59ab28f0)
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/main/main.c:2552
#17 0x0000000000cb836f in main (argc=2, argv=0x7ffc59ab2be8)
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/sapi/fpm/fpm/fpm_main.c:1966
some info that might be helpful...
(gdb) list 2667
2662 z = EX_CONSTANT(opline->op1);
2663
2664 if (Z_TYPE_P(z) == IS_STRING) {
2665 zend_string *str = Z_STR_P(z);
2666
2667 if (ZSTR_LEN(str) != 0) {
2668 zend_write(ZSTR_VAL(str), ZSTR_LEN(str));
2669 }
2670 } else {
2671 zend_string *str = _zval_get_string_func(z);
(gdb) print *z
$6 = {value = {lval = 140250487923552, dval = 6.9292947895499668e-310, counted = 0x7f8e9c832360, str
= 0x7f8e9c832360,
arr = 0x7f8e9c832360, obj = 0x7f8e9c832360, res = 0x7f8e9c832360, ref = 0x7f8e9c832360, ast =
0x7f8e9c832360, zv = 0x7f8e9c832360,
ptr = 0x7f8e9c832360, ce = 0x7f8e9c832360, func = 0x7f8e9c832360, ww = {w1 = 2625839968, w2 =
32654}}, u1 = {v = {type = 6 '\006',
type_flags = 0 '\000', const_flags = 0 '\000', reserved = 0
'\000'}, type_info = 6}, u2 = {next = 4294967295,
cache_slot = 4294967295, lineno = 4294967295, num_args = 4294967295, fe_pos = 4294967295,
fe_iter_idx = 4294967295,
access_flags = 4294967295, property_guard = 4294967295, extra = 4294967295}}
(gdb) print z->value->str
$7 = (zend_string *) 0x7f8e9c832360
(gdb) print *z->value->str
Cannot access memory at address 0x7f8e9c832360
------------------------------------------------------------------------
[2017-09-10 06:31:44] rasmus@php.net
Is this after a cache full event? As in, when you see this happening, what are the values of these
opcache vars in your phpinfo output?
OOM restarts
Hash keys restarts
Manual restarts
PHP 7.1.x has been running in production on a ton of heavily hit sites with at least some of them
making heavy use of memcached without ever seeing this. The ones I am involved with almost never hit
a cache reset condition, so that might be a difference.
------------------------------------------------------------------------
[2017-09-09 05:37:09] jjones at smugmug dot com
Since posting the previous core dump (and info extracted from it), I found the following in the
source.
ext/opcache/zend_file_cache.c: op_array~~>refcount = (uint32_t*)(intptr_t)-1;
Zend/zend_execute.c: #define ZEND_FAKE_OP_ARRAY ((zend_op_array*)(zend_intptr_t)-1)
which explains the 0xffffffff value of the "refcount" field in the failing call. So
scratch the question I asked about an intended decrement flipping a zero to all 0xff's.
------------------------------------------------------------------------
[2017-09-08 22:37:16] jjones at smugmug dot com
We're running a PHP7.1.9 build on our internal testing servers, preparing for a production
rollout to upgrade from PHP5.6 and are also seeing SegFaults.
Our setup is NGINX/php-fpm and the php-fpm processes sometimes running cleanly for non-trivial
amounts of time, then we'll see or more processes crash before stablizing again.
We have core dumps enabled, have build the binaries with optimization disabled, meaning
"-O0" for the C and C++ compiles. Ane we are running with
"opcache.protect_memory=1" already, since we were debugging another source of SegFaults
before this use case appeared.
Here's a backtrace from the most recent crash, with a few curious data points that may (or may
not?) be helpful.
This was the last of 5 core dumps in cluster today, so it's possible that it's the victim
of corrupted shared memory rather then the one that caused it in the first place. The previous core
dump (from a memcache extension issue) was generating core dumps too frequently to save them all.
So the current setup uses a static name for the core dumps, so the last one is all I have.
Backtrace:
#0 0x0000000000b64983 in zend_inline_hash_func (len=31426464, str=0x7fcf65800001 <error: Cannot
access memory at address 0x7fcf65800001>) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_string.h:331
#1 zend_hash_func (str=0x7fcf656e71a0 "", len=32577024) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_string.c:34
#2 0x0000000000b3bc9a in zend_string_hash_val (s=0x7fcf656e7188) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_string.h:86
#3 zend_hash_find_bucket (key=0x7fcf656e7188, ht=0x1ec34a0) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_hash.c:477
#4 zend_hash_find (ht=0x1ec34a0, key=0x7fcf656e7188) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_hash.c:1958
#5 0x00007fcf6536cbe5 in zend_accel_class_hash_copy (target=0x1ec34a0, source=0x7fced63878d0,
pCopyConstructor=0x7fcf6536b697 <zend_class_copy_ctor>) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/ext/opcache/zend_accelerator_util_funcs.c:572
#6 0x00007fcf6536d391 in zend_accel_load_script (persistent_script=0x7fced63877c0,
from_shared_memory=1) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/ext/opcache/zend_accelerator_util_funcs.c:655
#7 0x00007fcf6534ca87 in persistent_compile_file (file_handle=0x7fff7073a3b0, type=2) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/ext/opcache/ZendAccelerator.c:1948
#8 0x0000000000a987b6 in compile_filename (type=2, filename=0x7fcf65614790) at
Zend/zend_language_scanner.l:662
#9 0x0000000000ba94c9 in zend_include_or_eval (inc_filename=0x7fcf65614790, type=2) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_execute.c:2839
#10 0x0000000000c88862 in ZEND_INCLUDE_OR_EVAL_SPEC_TMPVAR_HANDLER () at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:51660
#11 0x0000000000ba9a6b in execute_ex (ex=0x7fcf656146a0) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#12 0x00007fcf635e77e0 in hp_execute_ex (execute_data=0x7fcf656146a0) at
/opt/xhprof/build/7.1.6/tideways.c:3603
#13 hp_execute_ex (execute_data=0x7fcf656146a0) at /opt/xhprof/build/7.1.6/tideways.c:3549
#14 0x0000000000aee90e in zend_call_function (fci=0x7fff7073ac50, fci_cache=0x7fff7073ac20) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_execute_API.c:855
#15 0x0000000000b4f153 in zend_call_method (object=0x0, obj_ce=0x0, fn_proxy=0x7fcf65663bd0,
function_name=0x7fcf65602a40 "__autoload", function_name_len=10, retval_ptr=0x0,
param_count=1, arg1=0x7fcf65614690, arg2=0x0)
at /home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_interfaces.c:99
#16 0x00000000008c3769 in zif_spl_autoload_call (execute_data=0x7fcf65614640,
return_value=0x7fff7073b260) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/ext/spl/php_spl.c:420
#17 0x0000000000ba612b in execute_internal (execute_data=0x7fcf65614640,
return_value=0x7fff7073b260) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_execute.c:2040
#18 0x0000000000aeea21 in zend_call_function (fci=0x7fff7073b2a0, fci_cache=0x7fff7073b270) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_execute_API.c:871
#19 0x0000000000aef8d3 in zend_lookup_class_ex (name=0x7fcf65663f90, key=0x0, use_autoload=1) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_execute_API.c:1028
#20 0x0000000000af07cf in zend_fetch_class (class_name=0x7fcf65663f90, fetch_type=512) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_execute_API.c:1441
#21 0x0000000000bb5cc1 in ZEND_FETCH_CLASS_SPEC_CV_HANDLER () at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:2444
#22 0x0000000000ba9a6b in execute_ex (ex=0x7fcf656145a0) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#23 0x00007fcf635e77e0 in hp_execute_ex (execute_data=0x7fcf656145a0) at
/opt/xhprof/build/7.1.6/tideways.c:3603
#24 hp_execute_ex (execute_data=0x7fcf656145a0) at /opt/xhprof/build/7.1.6/tideways.c:3549
#25 0x0000000000badfa4 in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:949
#26 0x0000000000ba9a6b in execute_ex (ex=0x7fcf656140e0) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#27 0x0000000000c88ec8 in ZEND_INCLUDE_OR_EVAL_SPEC_TMPVAR_HANDLER () at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:51699
#28 0x0000000000ba9a6b in execute_ex (ex=0x7fcf65614030) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:429
#29 0x0000000000baa42b in zend_execute (op_array=0x7fcf65672000, return_value=0x0) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend_vm_execute.h:474
#30 0x0000000000b0fcc1 in zend_execute_scripts (type=8, retval=0x0, file_count=3) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/Zend/zend.c:1480
#31 0x0000000000a5b290 in php_execute_script (primary_file=0x7fff7073dd60) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/main/main.c:2552
#32 0x0000000000cb836f in main (argc=2, argv=0x7fff7073e058) at
/home/jjones/ops/ops-tools/deb-build/work/php-7.1.9/sapi/fpm/fpm/fpm_main.c:1966
Crash caused by call to compute the hash of a null string with a bogus/corrupt length given.
Data structures/fields in the code mention in the stack that might be clues.
---- Bad "refcount" in string structure
(gdb) list
81 /*---*/
82
83 static zend_always_inline zend_ulong zend_string_hash_val(zend_string *s)
84 {
85 if (!ZSTR_H(s)) {
86 ZSTR_H(s) = zend_hash_func(ZSTR_VAL(s), ZSTR_LEN(s));
87 }
88 return ZSTR_H(s);
89 }
90
(gdb) print *s
$11 = {gc = {refcount = 4294967295, u = {v = {type = 0 '\000', flags = 0 '\000',
gc_info = 0}, type_info = 0}}, h = 0, len = 32577024, val = ""}
(gdb) printf "%x\n", s->gc->refcount
ffffffff
Meaning the "refcount" associated with the string is the maximum value that can fit in a
32-bit unsigned int. Or it's -1 in a signed 32-bit int. Maybe some refcount tracking code
overflowed the counter, or maybe a '0' value was decremented?
And finally, in case it helps, the functions call chain suggests PHP is crashing trying to fetch
something from cache, not while interpreting PHP code itself.
(gdb) print execute_data->func->common->function_name->val+0
$26 = 0x7fced3fb8248 "__autoload"
(gdb) print execute_data->prev_execute_data->func->common->function_name->val+0
$27 = 0x7fced3fc0be8 "spl_autoload_call"
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=71135
--
Edit this bug report at https://bugs.php.net/bug.php?id=71135&edit=1