Bug #80341 [Asn]: OPCache wasted memory grows rapidly and no restart after max_wasted_percentage

From: Date: Tue, 17 Nov 2020 10:51:09 +0000
Subject: Bug #80341 [Asn]: OPCache wasted memory grows rapidly and no restart after max_wasted_percentage
References: 1  Groups: php.bugs 
Request: Send a blank email to php-bugs+get-230403@lists.php.net to get a copy of this message
Edit report at https://bugs.php.net/bug.php?id=80341&edit=1

 ID:                 80341
 Updated by:         cmb@php.net
 Reported by:        alex at ndros dot com
 Summary:            OPCache wasted memory grows rapidly and no restart
                     after max_wasted_percentage
 Status:             Assigned
 Type:               Bug
 Package:            opcache
 Operating System:   Windows 2019
 PHP Version:        7.4.12
 Assigned To:        cmb
 Block user comment: N
 Private report:     N

 New Comment:

> […] , not regarding opcache.force_restart_timeout appears to be
> bug.

Well, on a closer look, it doesn't.  While on non Windows
platforms, all processes which share an OPcache instance are
forked childs of the same process, on Windows arbitrary processes
may share an OPcache instance, so forcefully terminating these
processes may not even be possible. At least, there is currently
no provision to track the relevant processes, so for now this is a
documentation problem, fixed with
<http://svn.php.net/viewvc?view=revision&revision=351404>.


Previous Comments:
------------------------------------------------------------------------
[2020-11-13 15:48:17] cmb@php.net

There are general issues regarding OPcache restarts on Windows.  A
long running process attached to the same OPcache instance
apparently completely blocks restarts, even if
opcache.force_restart_timeout has expired.  While a long running
request should probably use its own OPcache instance, not
regarding opcache.force_restart_timeout appears to be bug.

And if opcache.file_cache is enabled, restarts take considerable
time between the schedule and the actual restart, contrary to
opcache.file_cache being disabled.

> I cannot publish much more information here.

Okay, but actually I'm only interested in the "Reparse Tag Value"
of the SMB share, since we do not support arbitrary reparse tags,
and this might explain some issues.

------------------------------------------------------------------------
[2020-11-12 15:33:10] alex at ndros dot com

Hi there,

We're using SMB3. The network connection to the SMB share is perfectly stable.

The reason for the pre-mature updates was that OPcache stopped running altogether. All opcache
statistics froze. After wasted_memory filled my entire 2GB allocation, Opcache stopped running.

The wasted memory problem is gone since we removed filecache. Right now, Opcache is much healthier:

total memory: 256.00MB
used memory: 92.91MB
free memory: 163.09MB
wasted memory: 0.00b (0%)

number of cached files: 3,229
number of hits: 1,623,566,737
number of misses: 16,133
blacklist misses: 1,732,646
number of cached keys: 5,683
max cached keys: 130,987
interned strings usage
buffer size: 24.00MB
used memory: 7.59MB
free memory: 16.41MB
number of strings: 109,680

However, the problem that opcache will not reset after the max_wasted_percentage value is reached
still remains.

The output of that command is:

fsutil reparsePoint query D:\inetpub\wwwroot
Reparse Tag Value : 0xa000000c
Tag value: Microsoft
Tag value: Name Surrogate
Tag value: Symbolic Link

Reparse Data Length: 0x00000068
Reparse Data:
0000:  28 00 34 00 00 00 28 00  00 00 00 00 5c 00 5c 00  (.4...(.....\.\.
0010:  31 00 30 00 2e 00 31 00  2e 00 31 00 2e 00 31 00  1.0...1...1...1.
0020:  30 00 30 00 5c 00 77 00  77 00 77 00 72 00 6f 00  0.0.\.w.w.w.r.o.
0030:  6f 00 74 00 5c 00 3f 00  3f 00 5c 00 55 00 4e 00  o.t.\.?.?.\.U.N.
0040:  43 00 5c 00 31 00 30 00  2e 00 31 00 2e 00 31 00  C.\.1.0...1...1.
0050:  2e 00 31 00 30 00 30 00  5c 00 77 00 77 00 77 00  ..1.0.0.\.w.w.w.
0060:  72 00 6f 00 6f 00 74 00                           r.o.o.t.

I cannot publish much more information here. If you want, I can give you access to a VM in the load
balance, with the initial settings that caused the problem so that you can debug.

Thanks
Alex

------------------------------------------------------------------------
[2020-11-11 17:08:50] cmb@php.net

Also, which version of SMB is in use there?  Version 1 is insecure
and has other known issues, so you should use at least v2 or
better v3.

Furthermore, I'm assuming that the network connection to that SMB
share is very stable; otherwise an unstable connection might be
the root cause for "arbitrary" issues in this regard.  And likely,
you are better of deploying all the PHP files to the individual
machines instead of hosting them on an SMB share, anyway.

------------------------------------------------------------------------
[2020-11-11 16:13:20] cmb@php.net

Hmm, I still cannot reproduce any of the reported behavior.

> Yes, the files are located in a symlink pointing to an SMB
> share.

This might be reason for the misbehavior.  Could you please
post what

    fsutil reparsePoint query <path>

reports for the symlink and the SMB share?

Could you also please check whether realpath() resolves any of the
files on the SMB share correctly, and also what filemtime() and
stat()['mtime'] reports?  I'm not so much interested in the exact
timestamps, but rather whether these are "in the future" what could
explain the premature updates.

------------------------------------------------------------------------
[2020-11-09 16:05:02] cmb@php.net

Thanks for the swift reply!

I'll have a closer look.

------------------------------------------------------------------------


The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at

    https://bugs.php.net/bug.php?id=80341


--
Edit this bug report at https://bugs.php.net/bug.php?id=80341&edit=1


Thread (13 messages)

« previous php.bugs (#230403) next »