Edit report at https://bugs.php.net/bug.php?id=71135&edit=1
ID: 71135
Comment by: nick at noodles dot net dot nz
Reported by: iquito at gmx dot net
Summary: Random memory corruption with strings
Status: Closed
Type: Bug
Package: opcache
Operating System: Debian Jessie
PHP Version: 7.0.0
Assigned To: laruence
Block user comment: N
Private report: N
New Comment:
I can replicate the issue by rsync'ing our files to the production web servers over and over.
This causes the timestamps to be updated each time, opcache sees these as new files and stores them
again.
If I watch the wasted memory stats I can see that every time we rsync it increases by ~15MB. Once
the free memory is all used up I start to get memory corruption issues (bit flipping).
I'm using the defaults for opcache, so it should be firing opcache_reset when the wasted memory
is above 5%, but that doesn't look like it happens.
I'm having trouble keeping the server alive to get it to segfault when the memory corruption
occurs. As soon as I enable the memory protection flag I get segfaults every second.
This is the core backtrace (or at least what I can gather):
Core was generated by `/usr/sbin/httpd -DFOREGROUND'.
Program terminated with signal 11, Segmentation fault.
#0 0x00007f47054f6d43 in s_zval_to_payload.isra.21 () from /usr/lib64/php/modules/memcached.so
Missing separate debuginfos, use: debuginfo-install apr-1.4.8-3.el7.x86_64
apr-util-1.5.2-6.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-20.el7_2.x86_64
elfutils-libelf-0.166-2.el7.x86_64 elfutils-libs-0.166-2.el7.x86_64 enchant-1.6.0-8.el7.x86_64
expat-2.1.0-10.el7_3.x86_64 fastlz-0.1.0-0.1.20070619svnrev12.el7.x86_64
freetype-2.4.11-12.el7.x86_64 glib2-2.46.2-4.el7.x86_64 glibc-2.17-157.el7_3.5.x86_64
keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 libX11-1.6.3-3.el7.x86_64
libXau-1.0.8-2.1.el7.x86_64 libXpm-3.5.11-3.el7.x86_64 libattr-2.4.46-12.el7.x86_64
libcap-2.22-8.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libcurl-7.29.0-35.el7.centos.x86_64
libdb-5.3.21-19.el7.x86_64 libevent-2.0.21-4.el7.x86_64 libgcc-4.8.5-11.el7.x86_64
libgcrypt-1.5.3-13.el7_3.1.x86_64 libgpg-error-1.12-3.el7.x86_64 libicu-50.1.2-15.el7.x86_64
libidn-1.28-4.el7.x86_64 libjpeg-turbo-1.2.90-5.el7.x86_64 libmcrypt-2.5.8-13.el7.x86_64
libmemcached-libs-1.0.18-1.x86!
_64 libnghttp2-1.21.1-1.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 libselinux-2.5-6.el7.x86_64
libssh2-1.4.3-10.el7_2.1.x86_64 libstdc++-4.8.5-11.el7.x86_64 libtool-ltdl-2.4.2-22.el7_3.x86_64
libuuid-2.23.2-33.el7_3.2.x86_64 libwebp-0.3.0-3.el7.x86_64 libxcb-1.11-4.el7.x86_64
libxml2-2.9.1-6.el7_2.3.x86_64 libxslt-1.1.28-5.el7.x86_64 lua-5.1.4-15.el7.x86_64
mod_log_firstbyte-1.01-3.el7.x86_64 nspr-4.13.1-1.0.el7_3.x86_64 nss-3.28.4-1.2.el7_3.x86_64
nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64 nss-util-3.28.4-1.0.el7_3.x86_64
openldap-2.4.40-13.el7.x86_64 openssl-libs-1.0.1e-60.el7_3.1.x86_64 pcre-8.32-15.el7_2.1.x86_64
php-pecl-igbinary-2.0.1-1.el7.x86_64 php-pecl-memcached-3.0.3-1.el7.x86_64
php-pecl-msgpack-2.0.2-1.el7.x86_64 sqlite-3.7.17-8.el7.x86_64 systemd-libs-219-30.el7_3.9.x86_64
xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64
(gdb) bt
#0 0x00007f47054f6d43 in s_zval_to_payload.isra.21 () from /usr/lib64/php/modules/memcached.so
#1 0x00007f47054f7453 in s_memc_write_zval () from /usr/lib64/php/modules/memcached.so
#2 0x00007f47054f80ce in php_memc_store_impl.isra.25 () from /usr/lib64/php/modules/memcached.so
#3 0x00007f47123c1e6c in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at
/usr/src/debug/php-src-php-7.1.8/Zend/zend_vm_execute.h:970
#4 0x00007f471236ffbb in execute_ex (ex=<optimized out>) at
/usr/src/debug/php-src-php-7.1.8/Zend/zend_vm_execute.h:429
#5 0x00007f47123c3c74 in zend_execute (op_array=0x7f471167b000, op_array@entry=0x7f46fde42150,
return_value=return_value@entry=0x7f47116115f0)
at /usr/src/debug/php-src-php-7.1.8/Zend/zend_vm_execute.h:474
#6 0x00007f47123285b3 in zend_execute_scripts (type=type@entry=8, retval=0x7f47116115f0,
retval@entry=0x0, file_count=file_count@entry=3)
at /usr/src/debug/php-src-php-7.1.8/Zend/zend.c:1476
#7 0x00007f47122c5b18 in php_execute_script (primary_file=primary_file@entry=0x7ffd229e12a0) at
/usr/src/debug/php-src-php-7.1.8/main/main.c:2537
#8 0x00007f47123c5c7d in php_handler (r=<optimized out>) at
/usr/src/debug/php-src-php-7.1.8/sapi/apache2handler/sapi_apache2.c:712
#9 0x00007f47227a9550 in ap_run_handler (r=r@entry=0x7f47232b41b0) at config.c:170
#10 0x00007f47227a9a99 in ap_invoke_handler (r=r@entry=0x7f47232b41b0) at config.c:434
#11 0x00007f47227c05ea in ap_process_async_request (r=0x7f47232b41b0) at http_request.c:436
#12 0x00007f47227c08c4 in ap_process_request (r=r@entry=0x7f47232b41b0) at http_request.c:471
#13 0x00007f47227bc77d in ap_process_http_sync_connection (c=0x7f472338f7a0) at http_core.c:210
#14 ap_process_http_connection (c=0x7f472338f7a0) at http_core.c:251
#15 0x00007f47227b39c0 in ap_run_process_connection (c=c@entry=0x7f472338f7a0) at connection.c:42
#16 0x00007f47227b3f18 in ap_process_connection (c=c@entry=0x7f472338f7a0, csd=<optimized
out>) at connection.c:226
#17 0x00007f4714beaa50 in child_main (child_num_arg=child_num_arg@entry=0,
child_bucket=child_bucket@entry=0) at prefork.c:726
#18 0x00007f4714beacf1 in make_child (s=0x7f4722fb9420, slot=slot@entry=0, bucket=0) at
prefork.c:834
#19 0x00007f4714bead5f in startup_children (number_to_start=10) at prefork.c:853
#20 0x00007f4714bebc83 in prefork_run (_pconf=<optimized out>, plog=0x7f4722fbf3a8,
s=0x7f4722fb9420) at prefork.c:1020
#21 0x00007f472278b45e in ap_run_mpm (pconf=pconf@entry=0x7f4722f92188, plog=0x7f4722fbf3a8,
s=0x7f4722fb9420) at mpm_common.c:94
#22 0x00007f4722783d78 in main (argc=2, argv=0x7ffd229e19e8) at main.c:783
I'm not sure if this helps, but I can do more debugging if needed.
Previous Comments:
------------------------------------------------------------------------
[2017-08-28 23:03:30] nikic@php.net
If you are still experiencing this problem or something similar to it, please try the suggestion
from @laruence earlier in this thread: Enable the opcache.protect_memory ini option, which will
(likely) trigger a crash instead of silently corrupting the memory. It's hard to do anything
about this if we don't know where the invalid write occurs.
------------------------------------------------------------------------
[2017-08-28 22:52:47] jagoba at tapatalk dot com
We do have this problem too on production, during heavy loading we got errors about sql queries that
get mistery changed to "SELECT * FRPM" (notice the P), or links in our pages with www
changed to vww.
We got this issues both in php 7.1.8 and 7.1.3 with opcache, any hint or workaround is welcome.
------------------------------------------------------------------------
[2017-08-28 08:28:18] spam2 at rhsoft dot net
i face this below with 7.1.8 on the first production machine with 7.1.x randomly when under light
load shared files are updated - maybe related?
Sun Aug 20 19:45:38 2017 (3066): Warning Internal error: wrong size calculation:
global_mysql_ext.inc.php start=0x00007feba814c580, end=0x00007feba816fc60, real=0x00007feba816fc40
------------------------------------------------------------------------
[2017-08-28 00:06:51] nick at noodles dot net dot nz
I'm starting to see this problem on PHP 7.1.8. Seems the opcache is randomly being corrupted
one character at a time. This is on a high traffic website with multiple web servers.
------------------------------------------------------------------------
[2017-07-03 10:38:37] kalyanceg at gmail dot com
We are facing this issue on debian stretch with php7.0-opcache 7.0.19-1
We see opcache getting corrupted after some days of apache running continuously and gets fixed only
on restart of apache
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=71135
--
Edit this bug report at https://bugs.php.net/bug.php?id=71135&edit=1