[php-src] Issue #15539: OPCache File Inclusion and Function Call Errors: Letter Shifting Issue During High Load on PHP 8.3.10
| From: | sebastianthulin | Date: | Thu, 22 Aug 2024 09:21:29 +0000 |
| Subject: | [php-src] Issue #15539: OPCache File Inclusion and Function Call Errors: Letter Shifting Issue During High Load on PHP 8.3.10 | ||
| Groups: | php.bugs | ||
| Request: | Send a blank email to php-bugs+get-247574@lists.php.net to get a copy of this message | ||
Issue: https://github.com/php/php-src/issues/15539
Author: sebastianthulin
### Description
Hi,
I've been encountering a persistent and puzzling issue related to OPCache on a production
server running PHP 8.3.10 (though the issue has also occurred on PHP 8.2.x). I'm hoping to get
some insights or suggestions on why this error occurs.
Environment:
PHP Version: 8.3.10 (Issue also present in 8.2.x, a PHP update was made in order to mitigate the
error)
OPCache Version: 8.3.10
Web Server: LiteSpeed (with lsphp)
Caching: OPCache enabled
Traffic: High-traffic site
Memory: 120GB RAM (utilize max 15GB of these)
Disk: 1.5 TB (utilize 49GB)
Hosting: Openstack Compute
OS: Ubuntu 22.04.3 LTS (Jammy Jellyfish)
Number of vhosts: 2 (two nearly identical codebases are deployed in /var/www/prod/ and
/var/www/stage/)
Deployed solution: WordPress (highly adapted to usecase)
Problem Description:
About once a day, we encounter file inclusion errors where filenames and function names appear to be
"shifted" by one letter, leading to file not found or function does not exist errors.
Examples include:
- /includes/config.php being misread as /includes/comfig.php [The "n" in
"config" has been shifted to letter "m"]
- wp-includes/class-wp-textdomain-registry.php being misread as
wp-includes/class-wp-textdomain-registry.pho [The "p" in "php" has been shifted
to letter "p"]
- wp_get_current_user being misread as wp_get_current_urer() [The "s" in "user"
has been shifted to letter "r"].
Problem Pattern:
We have identified a very clear pattern here. It is on any file include or function call, and are
always one single letter that has been shifted to the left in the alphabet if an "b" in a
string is replaced, it will be the letter "a" that replaces the correct char.
These issues seem to be tied to periods of high traffic when many uncached requests hit the server
simultaneously (normally cached in pagecache). The problem resolves temporarily when PHP processes
are restarted, but it eventually recurs. In som cases the issue may occur in 10 minutes after a
reset, and in some cases up to 24 hours.
We have not concluded if this also affects variable names as this won't neccesarily result in a
fatal error (in most cases we would se a warning about undefined var).
Actions taken so far:
- Updated PHP from 8.2 to 8.3.10.
- Turned off JIT opcache.jit
- Increased JIT buffer size opcache.jit_buffer_size
- Turned off opcache.validate_timestamps
- Decreased / Increased opcache.revalidate_freq
- Increased opcache.max_accelerated_files
- Increasd opcache.interned_strings_buffer [Recommended in: https://bugs.php.net/bug.php?id=71135]
- Turned off opcache.enable_file_override
I have also checked the server for any other active php versions, that may run and affect the cache
and concluded that they are removed. We are only running lsphp 8.3.10.
Responses in this thread may be slow, due to the fact that the error is highly intermittent, please
allow some time for responses due to that changes needs at least 48 hours to be tested.
Configuration line:
'./configure' '--prefix=/usr/local/lsws/lsphp83' '--enable-cgi'
'--enable-cli' '--enable-phpdbg=no' '--enable-litespeed'
'--with-config-file-path=/usr/local/lsws/lsphp83/etc/php/8.3/litespeed/'
'--with-config-file-scan-dir=/usr/local/lsws/lsphp83/etc/php/8.3/mods-available/'
'--build=x86_64-linux-gnu' '--host=x86_64-linux-gnu'
'--libdir=${prefix}/lib/php' '--libexecdir=${prefix}/lib/php'
'--datadir=${prefix}/share/php/8.3' '--program-suffix=8.3'
'--disable-all' '--disable-debug' '--disable-rpath'
'--disable-static' '--with-pic' '--with-layout=GNU'
'--without-pear' '--enable-bcmath' '--enable-calendar'
'--enable-ctype' '--enable-dom' '--with-enchant'
'--enable-exif' '--with-gettext' '--with-gmp'
'--enable-fileinfo' '--enable-filter' '--enable-ftp'
'--enable-hash' '--with-iconv' '--enable-mbregex'
'--enable-mbregex-backtrack' '--enable-mbstring' '--enable-phar'
'--enable-posix' '--enable-mysqlnd-compression-support' '--with-zlib'
'--with-openssl=yes' '--with-libedit' '--with-libxml'
'--with-bz2' '--enable-session' '--enable-simplexml'
'--enable-soap' '--enable-sockets' '--enable-tokenizer'
'--enable-xml' '--enable-xmlreader' '--enable-xmlwriter'
'--with-xmlrpc' '--with-xsl' '--with-mhash=yes'
'--enable-sysvmsg' '--enable-sysvsem' '--enable-sysvshm'
'--with-zip' '--with-sodium' '--with-password-argon2'
'--with-system-tzdata' '--enable-gd' '--enable-gd-native-ttf'
'--with-jpeg' '--with-xpm' '--with-png' '--with-webp'
'--with-freetype' '--with-vpx-dir'
'--with-mysql-sock=/var/run/mysqld/mysqld.sock' '--disable-dtrace'
'--enable-pdo' '--enable-mysqlnd' '--enable-pcntl'
'--with-curl=shared,/usr' '--with-imap=shared,/usr' '--with-kerberos'
'--with-imap-ssl=yes' '--enable-intl=shared' '--with-ldap=shared,/usr'
'--with-ldap-sasl=/usr' '--with-mysqli=shared,mysqlnd'
'--with-pdo-mysql=shared,mysqlnd' '--with-pgsql=shared,/usr'
'--with-pdo-pgsql=shared,/usr' '--with-pspell=shared,/usr'
'--with-snmp=shared,/usr' '--with-sqlite3=shared,/usr'
'--with-pdo-sqlite=shared,/usr' '--with-pdo-dblib=shared,/usr'
'--with-tidy=shared,/usr' '--enable-opcache' '--enable-opcache-file'
'--enable-huge-code-pages' 'build_alias=x86_64-linux-gnu'
'host_alias=x86_64-linux-gnu' 'CFLAGS=-g -O2 -ffile-prefix-map=/build/php8.3-8.3.10=.
-flto=auto -ffat-lto-objects -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat
-Werror=format-security -O2 -Wall -fsigned-char -fno-strict-aliasing -fno-lto -g'
OPCache Directives:
opcache.enable = 1
opcache.enable_cli =
opcache.use_cwd = 1
opcache.validate_timestamps = 1
opcache.validate_permission =
opcache.validate_root = 1
opcache.dups_fix =
opcache.revalidate_path =
opcache.log_verbosity_level = 1
opcache.memory_consumption = 2147483648
opcache.interned_strings_buffer = 256
opcache.max_accelerated_files = 30000
opcache.max_wasted_percentage = 0.05
opcache.force_restart_timeout = 180
opcache.revalidate_freq = 10
opcache.preferred_memory_model =
opcache.blacklist_filename =
opcache.max_file_size = 0
opcache.error_log =
opcache.protect_memory =
opcache.save_comments = 1
opcache.record_warnings =
opcache.enable_file_override =
opcache.optimization_level = 2147401727
opcache.lockfile_path = /tmp
opcache.file_cache =
opcache.file_cache_only =
opcache.file_cache_consistency_checks = 1
opcache.file_update_protection = 2
opcache.opt_debug_level = 0
opcache.restrict_api =
opcache.huge_code_pages =
opcache.preload =
opcache.preload_user =
opcache.jit = tracing
opcache.jit_buffer_size = 1073741824
opcache.jit_debug = 0
opcache.jit_bisect_limit = 0
opcache.jit_blacklist_root_trace = 16
opcache.jit_blacklist_side_trace = 8
opcache.jit_hot_func = 127
opcache.jit_hot_loop = 64
opcache.jit_hot_return = 8
opcache.jit_hot_side_exit = 8
opcache.jit_max_exit_counters = 8192
opcache.jit_max_loop_unrolls = 8
opcache.jit_max_polymorphic_calls = 2
opcache.jit_max_recursive_calls = 2
opcache.jit_max_recursive_returns = 2
opcache.jit_max_root_traces = 1024
opcache.jit_max_side_traces = 128
opcache.jit_prof_threshold = 0
opcache.jit_max_trace_length = 1024
Interned strings buffer:
buffer_size = 268435456
used_memory = 83099352
free_memory = 185336104
number_of_strings = 229401
Opcache Status (newly restarted):
num_cached_scripts = 6295
num_cached_keys = 8895
max_cached_keys = 32531
hits = 28237442
start_time = 1724313429
last_restart_time = 0
oom_restarts = 0
hash_restarts = 0
manual_restarts = 0
misses = 8602
blacklist_misses = 0
blacklist_miss_ratio = 0
opcache_hit_rate = 99.969546177865
Resources found on similar issues:
- https://bugs.php.net/bug.php?id=71135
- https://github.com/php/php-src/issues/9373
### PHP Version
8.3.10
### Operating System
Ubuntu 22.04.3 LTS (Jammy Jellyfish)