Bug #71135 [Com]: Random memory corruption with strings

Edit report at https://bugs.php.net/bug.php?id=71135&edit=1 ID: 71135 Comment by: nick at noodles dot net dot nz Reported by: iquito at gmx dot net Summary: Random memory corruption with strings Status: Closed Type: Bug Package: opcache Operating System: Debian Jessie PHP Version: 7.0.0 Assigned To: laruence Block user comment: N Private report: N New Comment: The pecl-memcached patch helped (at least it's not segfaulting every second). But I'm still seeing random memory corruption. I managed to get some core dumps: Core was generated by `/usr/sbin/httpd -DFOREGROUND'. Program terminated with signal 11, Segmentation fault. #0 sljit_malloc_exec (size=816) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/pcrelib/sljit/sljitExecAllocator.c:196 196 header->prev_size = chunk_size; Missing separate debuginfos, use: debuginfo-install apr-1.4.8-3.el7.x86_64 apr-util-1.5.2-6.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 elfutils-libelf-0.166-2.el7.x86_64 elfutils-libs-0.166-2.el7.x86_64 enchant-1.6.0-8.el7.x86_64 expat-2.1.0-10.el7_3.x86_64 fastlz-0.1.0-0.1.20070619svnrev12.el7.x86_64 freetype-2.4.11-12.el7.x86_64 glib2-2.46.2-4.el7.x86_64 glibc-2.17-157.el7_3.5.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 libX11-1.6.3-3.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXpm-3.5.11-3.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libcurl-7.29.0-35.el7.centos.x86_64 libdb-5.3.21-19.el7.x86_64 libevent-2.0.21-4.el7.x86_64 libgcc-4.8.5-11.el7.x86_64 libgcrypt-1.5.3-13.el7_3.1.x86_64 libgpg-error-1.12-3.el7.x86_64 libicu-50.1.2-15.el7.x86_64 libidn-1.28-4.el7.x86_64 libjpeg-turbo-1.2.90-5.el7.x86_64 libmcrypt-2.5.8-13.el7.x86_64 libmemcached-libs-1.0.18-1.x86! _64 libnghttp2-1.21.1-1.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 libselinux-2.5-6.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_64 libstdc++-4.8.5-11.el7.x86_64 libtool-ltdl-2.4.2-22.el7_3.x86_64 libuuid-2.23.2-33.el7_3.2.x86_64 libwebp-0.3.0-3.el7.x86_64 libxcb-1.11-4.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 libxslt-1.1.28-5.el7.x86_64 lua-5.1.4-15.el7.x86_64 mod_log_firstbyte-1.01-3.el7.x86_64 nspr-4.13.1-1.0.el7_3.x86_64 nss-3.28.4-1.2.el7_3.x86_64 nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64 nss-util-3.28.4-1.0.el7_3.x86_64 openldap-2.4.40-13.el7.x86_64 openssl-libs-1.0.1e-60.el7_3.1.x86_64 pcre-8.32-15.el7_2.1.x86_64 php-pecl-igbinary-2.0.1-1.el7.x86_64 php-pecl-memcached-3.0.3-2.el7.x86_64 php-pecl-msgpack-2.0.2-1.el7.x86_64 sqlite-3.7.17-8.el7.x86_64 systemd-libs-219-30.el7_3.9.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64 (gdb) bt #0 sljit_malloc_exec (size=816) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/pcrelib/sljit/sljitExecAllocator.c:196 #1 sljit_generate_code (compiler=compiler@entry=0x7f228e3312f0) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/pcrelib/sljit/sljitNativeX86_common.c:466 #2 0x00007f227b988a50 in _pcre_jit_compile (re=re@entry=0x7f228e11e870, extra=extra@entry=0x7f228e2fc0f0, mode=mode@entry=0) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/pcrelib/pcre_jit_compile.c:10342 #3 0x00007f227b9639ed in php_pcre_study (external_re=external_re@entry=0x7f228e11e870, options=1, errorptr=errorptr@entry=0x7ffe7cbea888) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/pcrelib/pcre_study.c:1628 #4 0x00007f227b98b99b in pcre_get_compiled_regex_cache (regex=0x7f2266d00728) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/php_pcre.c:518 #5 0x00007f227b98e1a2 in php_pcre_replace (regex=<optimized out>, subject_str=subject_str@entry=0x7f227af11000, subject=subject@entry=0x7f227af11018 "<html>\n <head>\n \t<title>Gallstones</title>\n \t<meta name=\"originalfile\" content=\"AZ_d0198.xml\" />\n </head>\n <body>\n \n", ' ' <repeats 12 times>, "\n", ' ' <repeats 15 times>, "<h1>Gallstones </h1>\n", ' ' <repeats 15 times>, "\n "..., subject_len=12448, replace_val=replace_val@entry=0x7f227ae11830, is_callable_replace=is_callable_replace@entry=0, limit=limit@entry=-1, replace_count=replace_count@entry=0x7ffe7cbeaa5c) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/php_pcre.c:1132 #6 0x00007f227b98e2a2 in php_replace_in_subject (regex=regex@entry=0x7f227ae11820, replace=replace@entry=0x7f227ae11830, subject=subject@entry=0x7f227ae11840, limit=limit@entry=-1, is_callable_replace=is_callable_replace@entry=0, replace_count=replace_count@entry=0x7ffe7cbeaa5c) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/php_pcre.c:1495 #7 0x00007f227b98e6ae in preg_replace_impl (return_value=return_value@entry=0x7f227ae11770, regex=regex@entry=0x7f227ae11820, replace=0x7f227ae11830, subject=0x7f227ae11840, limit_val=-1, is_callable_replace=is_callable_replace@entry=0, is_filter=is_filter@entry=0) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/php_pcre.c:1554 #8 0x00007f227b98f20f in zif_preg_replace (execute_data=0x7f227ae117d0, return_value=0x7f227ae11770) at /usr/src/debug/php-src-php-7.1.8/ext/pcre/php_pcre.c:1593 #9 0x00007f227bb21006 in ZEND_DO_ICALL_SPEC_RETVAL_USED_HANDLER () at /usr/src/debug/php-src-php-7.1.8/Zend/zend_vm_execute.h:675 #10 0x00007f227bb10fbb in execute_ex (ex=<optimized out>) at /usr/src/debug/php-src-php-7.1.8/Zend/zend_vm_execute.h:429 #11 0x00007f227bb64c74 in zend_execute (op_array=0x7f227ae7a000, op_array@entry=0x7f22677811f0, return_value=return_value@entry=0x7f227ae11030) at /usr/src/debug/php-src-php-7.1.8/Zend/zend_vm_execute.h:474 #12 0x00007f227bac95b3 in zend_execute_scripts (type=type@entry=8, retval=0x7f227ae11030, retval@entry=0x0, file_count=file_count@entry=3) at /usr/src/debug/php-src-php-7.1.8/Zend/zend.c:1476 #13 0x00007f227ba66b18 in php_execute_script (primary_file=primary_file@entry=0x7ffe7cbece90) at /usr/src/debug/php-src-php-7.1.8/main/main.c:2537 #14 0x00007f227bb66c7d in php_handler (r=<optimized out>) at /usr/src/debug/php-src-php-7.1.8/sapi/apache2handler/sapi_apache2.c:712 #15 0x00007f228bf4a550 in ap_run_handler (r=r@entry=0x7f228e1f5968) at config.c:170 #16 0x00007f228bf4aa99 in ap_invoke_handler (r=r@entry=0x7f228e1f5968) at config.c:434 #17 0x00007f228bf6094c in ap_internal_redirect (new_uri=<optimized out>, r=<optimized out>) at http_request.c:765 #18 0x00007f2280e49f1c in handler_redirect (r=0x7f228e1af910) at mod_rewrite.c:5195 #19 0x00007f228bf4a550 in ap_run_handler (r=r@entry=0x7f228e1af910) at config.c:170 #20 0x00007f228bf4aa99 in ap_invoke_handler (r=r@entry=0x7f228e1af910) at config.c:434 #21 0x00007f228bf615ea in ap_process_async_request (r=0x7f228e1af910) at http_request.c:436 #22 0x00007f228bf618c4 in ap_process_request (r=r@entry=0x7f228e1af910) at http_request.c:471 #23 0x00007f228bf5d77d in ap_process_http_sync_connection (c=0x7f228e191310) at http_core.c:210 #24 ap_process_http_connection (c=0x7f228e191310) at http_core.c:251 #25 0x00007f228bf549c0 in ap_run_process_connection (c=c@entry=0x7f228e191310) at connection.c:42 #26 0x00007f228bf54f18 in ap_process_connection (c=c@entry=0x7f228e191310, csd=<optimized out>) at connection.c:226 #27 0x00007f227e38ba50 in child_main (child_num_arg=child_num_arg@entry=10, child_bucket=child_bucket@entry=0) at prefork.c:726 #28 0x00007f227e38bcf1 in make_child (s=0x7f228ddbd420, slot=10, bucket=0) at prefork.c:834 #29 0x00007f227e38cc34 in perform_idle_server_maintenance (p=<optimized out>) at prefork.c:942 #30 prefork_run (_pconf=<optimized out>, plog=<optimized out>, s=<optimized out>) at prefork.c:1138 #31 0x00007f228bf2c45e in ap_run_mpm (pconf=pconf@entry=0x7f228dd96188, plog=0x7f228ddc33a8, s=0x7f228ddbd420) at mpm_common.c:94 #32 0x00007f228bf24d78 in main (argc=2, argv=0x7ffe7cbed658) at main.c:783 Hopefully this helps Previous Comments: ------------------------------------------------------------------------ [2017-08-29 02:52:17] nick at noodles dot net dot nz For reference, this seems to be related to pecl-memcached and interned strings. This patch worked for me: https://github.com/php-memcached-dev/php-memcached/commit/5f28025c15c87d0895f39def77068c8fd66d442a ------------------------------------------------------------------------ [2017-08-29 00:40:02] nick at noodles dot net dot nz I can replicate the issue by rsync'ing our files to the production web servers over and over. This causes the timestamps to be updated each time, opcache sees these as new files and stores them again. If I watch the wasted memory stats I can see that every time we rsync it increases by ~15MB. Once the free memory is all used up I start to get memory corruption issues (bit flipping). I'm using the defaults for opcache, so it should be firing opcache_reset when the wasted memory is above 5%, but that doesn't look like it happens. I'm having trouble keeping the server alive to get it to segfault when the memory corruption occurs. As soon as I enable the memory protection flag I get segfaults every second. This is the core backtrace (or at least what I can gather): Core was generated by `/usr/sbin/httpd -DFOREGROUND'. Program terminated with signal 11, Segmentation fault. #0 0x00007f47054f6d43 in s_zval_to_payload.isra.21 () from /usr/lib64/php/modules/memcached.so Missing separate debuginfos, use: debuginfo-install apr-1.4.8-3.el7.x86_64 apr-util-1.5.2-6.el7.x86_64 bzip2-libs-1.0.6-13.el7.x86_64 cyrus-sasl-lib-2.1.26-20.el7_2.x86_64 elfutils-libelf-0.166-2.el7.x86_64 elfutils-libs-0.166-2.el7.x86_64 enchant-1.6.0-8.el7.x86_64 expat-2.1.0-10.el7_3.x86_64 fastlz-0.1.0-0.1.20070619svnrev12.el7.x86_64 freetype-2.4.11-12.el7.x86_64 glib2-2.46.2-4.el7.x86_64 glibc-2.17-157.el7_3.5.x86_64 keyutils-libs-1.5.8-3.el7.x86_64 krb5-libs-1.14.1-27.el7_3.x86_64 libX11-1.6.3-3.el7.x86_64 libXau-1.0.8-2.1.el7.x86_64 libXpm-3.5.11-3.el7.x86_64 libattr-2.4.46-12.el7.x86_64 libcap-2.22-8.el7.x86_64 libcom_err-1.42.9-9.el7.x86_64 libcurl-7.29.0-35.el7.centos.x86_64 libdb-5.3.21-19.el7.x86_64 libevent-2.0.21-4.el7.x86_64 libgcc-4.8.5-11.el7.x86_64 libgcrypt-1.5.3-13.el7_3.1.x86_64 libgpg-error-1.12-3.el7.x86_64 libicu-50.1.2-15.el7.x86_64 libidn-1.28-4.el7.x86_64 libjpeg-turbo-1.2.90-5.el7.x86_64 libmcrypt-2.5.8-13.el7.x86_64 libmemcached-libs-1.0.18-1.x86! _64 libnghttp2-1.21.1-1.el7.x86_64 libpng-1.5.13-7.el7_2.x86_64 libselinux-2.5-6.el7.x86_64 libssh2-1.4.3-10.el7_2.1.x86_64 libstdc++-4.8.5-11.el7.x86_64 libtool-ltdl-2.4.2-22.el7_3.x86_64 libuuid-2.23.2-33.el7_3.2.x86_64 libwebp-0.3.0-3.el7.x86_64 libxcb-1.11-4.el7.x86_64 libxml2-2.9.1-6.el7_2.3.x86_64 libxslt-1.1.28-5.el7.x86_64 lua-5.1.4-15.el7.x86_64 mod_log_firstbyte-1.01-3.el7.x86_64 nspr-4.13.1-1.0.el7_3.x86_64 nss-3.28.4-1.2.el7_3.x86_64 nss-softokn-freebl-3.16.2.3-14.4.el7.x86_64 nss-util-3.28.4-1.0.el7_3.x86_64 openldap-2.4.40-13.el7.x86_64 openssl-libs-1.0.1e-60.el7_3.1.x86_64 pcre-8.32-15.el7_2.1.x86_64 php-pecl-igbinary-2.0.1-1.el7.x86_64 php-pecl-memcached-3.0.3-1.el7.x86_64 php-pecl-msgpack-2.0.2-1.el7.x86_64 sqlite-3.7.17-8.el7.x86_64 systemd-libs-219-30.el7_3.9.x86_64 xz-libs-5.2.2-1.el7.x86_64 zlib-1.2.7-17.el7.x86_64 (gdb) bt #0 0x00007f47054f6d43 in s_zval_to_payload.isra.21 () from /usr/lib64/php/modules/memcached.so #1 0x00007f47054f7453 in s_memc_write_zval () from /usr/lib64/php/modules/memcached.so #2 0x00007f47054f80ce in php_memc_store_impl.isra.25 () from /usr/lib64/php/modules/memcached.so #3 0x00007f47123c1e6c in ZEND_DO_FCALL_SPEC_RETVAL_UNUSED_HANDLER () at /usr/src/debug/php-src-php-7.1.8/Zend/zend_vm_execute.h:970 #4 0x00007f471236ffbb in execute_ex (ex=<optimized out>) at /usr/src/debug/php-src-php-7.1.8/Zend/zend_vm_execute.h:429 #5 0x00007f47123c3c74 in zend_execute (op_array=0x7f471167b000, op_array@entry=0x7f46fde42150, return_value=return_value@entry=0x7f47116115f0) at /usr/src/debug/php-src-php-7.1.8/Zend/zend_vm_execute.h:474 #6 0x00007f47123285b3 in zend_execute_scripts (type=type@entry=8, retval=0x7f47116115f0, retval@entry=0x0, file_count=file_count@entry=3) at /usr/src/debug/php-src-php-7.1.8/Zend/zend.c:1476 #7 0x00007f47122c5b18 in php_execute_script (primary_file=primary_file@entry=0x7ffd229e12a0) at /usr/src/debug/php-src-php-7.1.8/main/main.c:2537 #8 0x00007f47123c5c7d in php_handler (r=<optimized out>) at /usr/src/debug/php-src-php-7.1.8/sapi/apache2handler/sapi_apache2.c:712 #9 0x00007f47227a9550 in ap_run_handler (r=r@entry=0x7f47232b41b0) at config.c:170 #10 0x00007f47227a9a99 in ap_invoke_handler (r=r@entry=0x7f47232b41b0) at config.c:434 #11 0x00007f47227c05ea in ap_process_async_request (r=0x7f47232b41b0) at http_request.c:436 #12 0x00007f47227c08c4 in ap_process_request (r=r@entry=0x7f47232b41b0) at http_request.c:471 #13 0x00007f47227bc77d in ap_process_http_sync_connection (c=0x7f472338f7a0) at http_core.c:210 #14 ap_process_http_connection (c=0x7f472338f7a0) at http_core.c:251 #15 0x00007f47227b39c0 in ap_run_process_connection (c=c@entry=0x7f472338f7a0) at connection.c:42 #16 0x00007f47227b3f18 in ap_process_connection (c=c@entry=0x7f472338f7a0, csd=<optimized out>) at connection.c:226 #17 0x00007f4714beaa50 in child_main (child_num_arg=child_num_arg@entry=0, child_bucket=child_bucket@entry=0) at prefork.c:726 #18 0x00007f4714beacf1 in make_child (s=0x7f4722fb9420, slot=slot@entry=0, bucket=0) at prefork.c:834 #19 0x00007f4714bead5f in startup_children (number_to_start=10) at prefork.c:853 #20 0x00007f4714bebc83 in prefork_run (_pconf=<optimized out>, plog=0x7f4722fbf3a8, s=0x7f4722fb9420) at prefork.c:1020 #21 0x00007f472278b45e in ap_run_mpm (pconf=pconf@entry=0x7f4722f92188, plog=0x7f4722fbf3a8, s=0x7f4722fb9420) at mpm_common.c:94 #22 0x00007f4722783d78 in main (argc=2, argv=0x7ffd229e19e8) at main.c:783 I'm not sure if this helps, but I can do more debugging if needed. ------------------------------------------------------------------------ [2017-08-28 23:03:30] nikic@php.net If you are still experiencing this problem or something similar to it, please try the suggestion from @laruence earlier in this thread: Enable the opcache.protect_memory ini option, which will (likely) trigger a crash instead of silently corrupting the memory. It's hard to do anything about this if we don't know where the invalid write occurs. ------------------------------------------------------------------------ [2017-08-28 22:52:47] jagoba at tapatalk dot com We do have this problem too on production, during heavy loading we got errors about sql queries that get mistery changed to "SELECT * FRPM" (notice the P), or links in our pages with www changed to vww. We got this issues both in php 7.1.8 and 7.1.3 with opcache, any hint or workaround is welcome. ------------------------------------------------------------------------ [2017-08-28 08:28:18] spam2 at rhsoft dot net i face this below with 7.1.8 on the first production machine with 7.1.x randomly when under light load shared files are updated - maybe related? Sun Aug 20 19:45:38 2017 (3066): Warning Internal error: wrong size calculation: global_mysql_ext.inc.php start=0x00007feba814c580, end=0x00007feba816fc60, real=0x00007feba816fc40 ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=71135 -- Edit this bug report at https://bugs.php.net/bug.php?id=71135&edit=1