Bug #72645 [Ver]: php failed with 500 error 0xfffffffe, "Cannot create mutex" in opcache.log

Edit report at https://bugs.php.net/bug.php?id=72645&edit=1 ID: 72645 Updated by: ab@php.net Reported by: pvasilevich at plesk dot com Summary: php failed with 500 error 0xfffffffe, "Cannot create mutex" in opcache.log Status: Verified Type: Bug Package: opcache Operating System: Windows PHP Version: 7.0.9 Block user comment: N Private report: N New Comment: I think I've more clearance on this case now, and hopefully more understanding :) Namely, I can only reproduce this behavior when using an account with certain privileges as a pool identity. When I use a non elevated custom account, or a built-in LocalService or NetworkService - this is reproducable. When I use an account with the admin membership, or built-in LocalSystem - everything works fine. Same when using AppPoolIdentity. There is also an interesting piece of reading here http://www.iis.net/learn/manage/configuring-security/application-pool-identities http://adopenstatic.com/cs/blogs/ken/archive/2008/01/29/15759.aspx [quote] On Windows 7 and Windows Server 2008 R2, and later versions of Windows, the default is to run application pools as the application pool identity. To make this happen, a new identity type with the name "AppPoolIdentity" was introduced. If the "AppPoolIdentity" identity type is selected (the default on Windows 7 and Windows Server 2008 R2, and later), IIS will run worker processes as the application pool identity. With every other identity type, the security identifier will only be injected into the access token of the process. If the identifier is injected, content can still be ACLed for the ApplicationPoolIdentity, but the owner of the token is probably not unique. Here is an article that explains this concept. [/quote] This is actually matching with what I see in the practice. Fe i set both pools to run under the AppPoolIdentity, then this is what i see in the procexp \BaseNamedObjects\ZendOPcache.SharedMemoryMutex@pool0@943d85cdd3a3751784e90ca8856128b0 \BaseNamedObjects\ZendOPcache.SharedMemoryArea@pool0@943d85cdd3a3751784e90ca8856128b0 \BaseNamedObjects\ZendOPcache.SharedMemoryMutex@pool1@943d85cdd3a3751784e90ca8856128b0 \BaseNamedObjects\ZendOPcache.SharedMemoryArea@pool1@943d85cdd3a3751784e90ca8856128b0 And this is perfectly fine. Both the mutex and the shared memory are separated. It was a bit confusing when you told same about same address is written into the base address file. But it actually doesn't matter - both the mutex and the memory handles are named. Processes, that attach to the shared memory, may indeed use same virtual address. However - the named handles do indeed point to the different memory chunks. So in the case of using the AppPoolIdentity everything is separated. IMHO it is a nice concept which works correct out of the box. Now, to the case of using same identity for the different pools. Here's the chunk of reading that applies to the built-in accounts https://technet.microsoft.com/library/hh831797.aspx#ApplicationPoolIdentity LocalService and NetworkService are under the user group limitation. LocalSystem is in admins. This explains what is actually going on also with a custom pool identity. The app poll, and consequently the PHP process, will be started with under the configured custom account. If the underlying account doesn't have sufficient privilege, the injected pool identity will override the ACLs. OFC the more privileges an account has, the more security implications arise. My conclusion on this is, that most likely no action is required. Using AppPoolIdentity will perfectly fine separate the apps when using one app per pool. On the other hand, if different pools need to share same cache, they need to have same identity with sufficient privileges. Thus, both scenarios are possible with an appropriate server setup. In any case, the proposed patch has the same effect as if the default setting were used. And any manipulations with the security descriptors won't override the system user/group policies. I would most likely say this behavior is expected and is not a bug. Thanks. Previous Comments: ------------------------------------------------------------------------ [2016-07-26 12:49:18] ab@php.net Thanks for the further investigation, Pavel. The fact processes under different pools both attach to the same memory, while using different mutexes, is dangerous. That locking functionality is used also for various cache operations, so race conditions are to expect. If we go by separate mutex, it has to be separate memory. Yeah, the cache per app is a known crunch point regarding Opcache and shared hosting. Not Windows only, but fe on Linux with FPM. The ASLR limitation, and issues with shared memory are typical to Windows, on the other side. While ASLR is a real black box, it is somewhat easier with shared memory. On 64-bit and with a smaller cache size, memory issues are less probable. But also to mention is the file cache - it can not only help to separate sites, but also to wokaround ASLR issues. You're right, the primary security token won't work, as it is the exact issue we're talking about. I guess some impersonation still happens if the specific IIS configuration is chosen. And with the current implementation, the token used for both CreateMutex and CreateFilemapping comes from the impersonation context. Thanks. ------------------------------------------------------------------------ [2016-07-26 05:56:42] pvasilevich at plesk dot com >What i'm only concerned about is, that this would indeed split all the cache. Other >variants like mmap or SHM don't do this. I guess we could solve this by setting corresponding >security attributes. Checking currently impacts and possibilities. What I see in fact that both processes (running under the same user but in different pools) have written the same base address in their separate files (in my build of PHP). So mutex only protects exclusive access to the file, file contains memory address, and address is the same, so there is no cache split in fact. From other side I think currently concept of opcache cannot be used per application. I would like to have cache explicitly dedicated for my application with my memory limit I have specified and tuned for my application. In the same time, I want to have another cache for another application, that is working on the same server. I don't want to share any data between these applications. I don't want to share memory between these applications. I can try to configure this by having 2 different php.ini files for both my applications and explicitly selected mmap_base addresses in php.ini (guessing these addresses), but this is not user friendly to configure and you have risks that some DLLs with ASLR will be overlapped with selected address. There are things to improve in future versions. Regarding current situation, you can try to fix this by explicitly specifying Security Attributes structure when creating Mutex, but in this case you need to fetch SID of current user, and this should be not primary token (which probably will be app_pool_name), but need to resolve SID of user based on GetUserName(). > What is fastcgi.impersonate set to in your INI? > Would you include your INI configuration? I have tried both fastcgi.impersonate = 0 and fastcgi.impersonate = 1 the problem is the same my php.ini is default php.ini-production from php-7.0.9 with zend_extension=D:\php-7.0.9\ext\php_opcache.dll opcache.error_log=c:\windows\temp\opcache.log no other changes. ------------------------------------------------------------------------ [2016-07-25 18:32:29] mattficken@php.net What is fastcgi.impersonate set to in your INI? Would you include your INI configuration? ------------------------------------------------------------------------ [2016-07-25 16:14:37] ab@php.net Further info - it looks like phptest owns the mutex, but it's not listed by procexp explicitly. Thanks. ------------------------------------------------------------------------ [2016-07-25 15:33:51] ab@php.net Thanks for the report and analysis. Seems you also come to the root cause of the bug #72332 :) By the current implementation, the first request served will create the mutex and become its owner. With your finding it's clear, that another app pool will fail to create a mutex with the same name. It is currently unclear, which other implications such a configuration might have. It seems, that using the identity config other than app pool will produce this issue. Probably just changing its name is the simplest solution. What i'm only concerned about is, that this would indeed split all the cache. Other variants like mmap or SHM don't do this. I guess we could solve this by setting corresponding security attributes. Checking currently impacts and possibilities. Thanks. ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=72645 -- Edit this bug report at https://bugs.php.net/bug.php?id=72645&edit=1