Bug #80341 [Asn]: OPCache wasted memory grows rapidly and no restart after max_wasted_percentage

Edit report at https://bugs.php.net/bug.php?id=80341&edit=1 ID: 80341 Updated by: cmb@php.net Reported by: alex at ndros dot com Summary: OPCache wasted memory grows rapidly and no restart after max_wasted_percentage Status: Assigned Type: Bug Package: opcache Operating System: Windows 2019 PHP Version: 7.4.12 Assigned To: cmb Block user comment: N Private report: N New Comment: There are general issues regarding OPcache restarts on Windows. A long running process attached to the same OPcache instance apparently completely blocks restarts, even if opcache.force_restart_timeout has expired. While a long running request should probably use its own OPcache instance, not regarding opcache.force_restart_timeout appears to be bug. And if opcache.file_cache is enabled, restarts take considerable time between the schedule and the actual restart, contrary to opcache.file_cache being disabled. > I cannot publish much more information here. Okay, but actually I'm only interested in the "Reparse Tag Value" of the SMB share, since we do not support arbitrary reparse tags, and this might explain some issues. Previous Comments: ------------------------------------------------------------------------ [2020-11-12 15:33:10] alex at ndros dot com Hi there, We're using SMB3. The network connection to the SMB share is perfectly stable. The reason for the pre-mature updates was that OPcache stopped running altogether. All opcache statistics froze. After wasted_memory filled my entire 2GB allocation, Opcache stopped running. The wasted memory problem is gone since we removed filecache. Right now, Opcache is much healthier: total memory: 256.00MB used memory: 92.91MB free memory: 163.09MB wasted memory: 0.00b (0%) number of cached files: 3,229 number of hits: 1,623,566,737 number of misses: 16,133 blacklist misses: 1,732,646 number of cached keys: 5,683 max cached keys: 130,987 interned strings usage buffer size: 24.00MB used memory: 7.59MB free memory: 16.41MB number of strings: 109,680 However, the problem that opcache will not reset after the max_wasted_percentage value is reached still remains. The output of that command is: fsutil reparsePoint query D:\inetpub\wwwroot Reparse Tag Value : 0xa000000c Tag value: Microsoft Tag value: Name Surrogate Tag value: Symbolic Link Reparse Data Length: 0x00000068 Reparse Data: 0000: 28 00 34 00 00 00 28 00 00 00 00 00 5c 00 5c 00 (.4...(.....\.\. 0010: 31 00 30 00 2e 00 31 00 2e 00 31 00 2e 00 31 00 1.0...1...1...1. 0020: 30 00 30 00 5c 00 77 00 77 00 77 00 72 00 6f 00 0.0.\.w.w.w.r.o. 0030: 6f 00 74 00 5c 00 3f 00 3f 00 5c 00 55 00 4e 00 o.t.\.?.?.\.U.N. 0040: 43 00 5c 00 31 00 30 00 2e 00 31 00 2e 00 31 00 C.\.1.0...1...1. 0050: 2e 00 31 00 30 00 30 00 5c 00 77 00 77 00 77 00 ..1.0.0.\.w.w.w. 0060: 72 00 6f 00 6f 00 74 00 r.o.o.t. I cannot publish much more information here. If you want, I can give you access to a VM in the load balance, with the initial settings that caused the problem so that you can debug. Thanks Alex ------------------------------------------------------------------------ [2020-11-11 17:08:50] cmb@php.net Also, which version of SMB is in use there? Version 1 is insecure and has other known issues, so you should use at least v2 or better v3. Furthermore, I'm assuming that the network connection to that SMB share is very stable; otherwise an unstable connection might be the root cause for "arbitrary" issues in this regard. And likely, you are better of deploying all the PHP files to the individual machines instead of hosting them on an SMB share, anyway. ------------------------------------------------------------------------ [2020-11-11 16:13:20] cmb@php.net Hmm, I still cannot reproduce any of the reported behavior. > Yes, the files are located in a symlink pointing to an SMB > share. This might be reason for the misbehavior. Could you please post what fsutil reparsePoint query <path> reports for the symlink and the SMB share? Could you also please check whether realpath() resolves any of the files on the SMB share correctly, and also what filemtime() and stat()['mtime'] reports? I'm not so much interested in the exact timestamps, but rather whether these are "in the future" what could explain the premature updates. ------------------------------------------------------------------------ [2020-11-09 16:05:02] cmb@php.net Thanks for the swift reply! I'll have a closer look. ------------------------------------------------------------------------ [2020-11-09 14:36:21] alex at ndros dot com Hi there, Yes, I'm using php-7.4.12-nts-Win32-vc15-x64.zip We're using a load balance with multiple IIS server. Yes, the files are located in a symlink pointing to an SMB share. But it seems that I have solved the problem with the wasted memory. I switched off opcache.file_cache and now after 12 hours the wasted memory is at 0%. So whatever it was, it was somehow related to the file_cache. However, the problem that Opcache not reseting after opcache.max_wasted_percentage is reached still remains. I did several tests for this, and it never resets when the value was reached. ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=80341 -- Edit this bug report at https://bugs.php.net/bug.php?id=80341&edit=1