Bug #80814 [Csd]: threaded mod_php won't load: No space available for static Thread Local Storage

Edit report at https://bugs.php.net/bug.php?id=80814&edit=1 ID: 80814 Updated by: dmitry@php.net Reported by: theultramage at gmail dot com Summary: threaded mod_php won't load: No space available for static Thread Local Storage Status: Closed Type: Bug Package: Dynamic loading Operating System: FreeBSD 12.2 PHP Version: 8.0.2 Assigned To: dmitry Block user comment: N Private report: N New Comment: @alien I think, the problem may occur because Apache loads few modules that use static TLS and the required amount exceeds the available size. (It may be increased. See https://www.gnu.org/software/libc/manual/html_node/Dynamic-Linking-Tunables.html). But this would cause failure on each restart. May be the problem is somehow related to graceful restart. May be some Apache modules, or shared libraries were updated between restarts and this somehow triggered the failure. Let me know, if you find more info. Previous Comments: ------------------------------------------------------------------------ [2022-01-02 10:17:22] alien at rmail dot be I'm on linux with version 8.0.5 and I've noticed that logrotate, which does a httpd reload, which is a graceful restart, cannot load modphp (sometimes), due to what appears to be exactly this problem... maybe not only freeBSD has trouble with this kind of thing... The problem is that i had it 1st of Nov and 1st of Jan this year, but not on december? so it does not happen always, not easily reproducable, but it's still twice: [Sat Jan 01 04:02:01.476121 2022] [mpm_prefork:notice] [pid 1060] AH00171: Graceful restart requested, doing restart httpd: Syntax error on line 54 of /etc/httpd/conf/httpd.conf: Syntax error on line 1 of /etc/httpd/conf/modules.d/70_mod_php.conf: Cannot load modules/mod_php.so into server: /etc/httpd/modules/mod_php.so: cannot allocate memory in static TLS block ------------------------------------------------------------------------ [2021-03-10 13:04:47] dmitry@php.net Automatic comment on behalf of dmitry@zend.com Revision: http://git.php.net/?p=php-src.git;a=commit;h=3b377b51a22681f4594f8eb55e6de25ea01204c1 Log: Fixed bug #80814 (threaded mod_php won't load on FreeBSD: No space available for static Thread Local Storage) ------------------------------------------------------------------------ [2021-03-09 07:44:11] dmitry@php.net The code in TSRM.c allows to use the most efficient "local-exec" access model from JIT-ed code. If tsrm_get_ls_cache_tcb_offset() returns 0, zend_jit_setup() switches JIT code-generator to less efficient access method. This works fine on Linux with both GCC and CLANG. I didn't get, if you tested the patch, and it's good enough, to fix the problem on FreeBSD. If it misses something (e.g. i386 support) please extend it. ------------------------------------------------------------------------ [2021-03-05 12:59:15] theultramage at gmail dot com Ah. I see. Your comment showed that the situation is trickier than I originally considered. I will try to clarify things based on what I've learned so far. tls_model("initial-exec") is meant to be used in code whose memory layout can be resolved during program initialization. So ordinary executables, static .a libraries, or dynamic .so modules which are hard-bound to the main executable via import table. It's not meant for true dynamic libraries, loaded via dlopen(). Or so I've read. OSes do seem to have a way of making this work anyway, in a limited manner. FreeBSD prepares 128 bytes of space in every process. I also tested Ubuntu 20, and found that the limit was 1720 bytes. I do not know if this is just an improvised compat mechanism, or if it is something documented and fully endorsed by the ABI specification. I have not consulted this with core devs on the mailing list yet. The opinion from #clang is that this should not be used. The limit is global for the whole process. If each module needs 8 bytes, then at most 16 such modules can be loaded. Currently I am counting 33 php extensions that actively use ZEND_TSRMLS_CACHE_DEFINE() (and 18 that #include TSRM.h but don't use it?), so I don't believe it's currently possible to have them all dynamically loaded on FreeBSD. Though that is not a good enough reason to disable the optimization entirely, if it indeed leads to observable performance improvement. I believe that for cli/cgi/fpm, a workaround would be to have a bunch of the extensions compiled-in. But then again, I'm realizing that these three appear to be single-threaded static executables that do not need ZTS or thread local variables, so non of this matters for them. The other problem that I described (libphp.so requiring 328 bytes of static tls storage) thus turns out to be limited to mod_php, which is a dynamically loaded shared library. The reason is that if any piece of code requires the "initial-exec" static tls model, then the whole shared library is flagged as such. Here, not only does zend.c's use of TSRMLS_CACHE do it, but so will any threaded extension that gets compiled in. Furthermore, the mere presence of 'gottpoff' (and perhaps 'ntpoff') in inline assembly code will force static tls mode. Regarding your patch, it looks very close to what I arrived at on my end. For opcache I also patched out the i386 branch just in case. It makes me wonder though, why does tsrm_get_ls_cache_tcb_offset() in TSRM.c even exist, when opcache completely replicates it? An alternative way, that keeps the extensions as STATIC_TLS, would be to remove that assembly code from TSRM.c, and disable that TSRM_TLS_MODEL_ATTR on libphp.so itself and any extensions that get compiled in, while still applying it on shared extensions somehow. ------------------------------------------------------------------------ [2021-03-05 09:28:19] dmitry@php.net https://github.com/php/php-src/commit/2aefd112114bd150f5dbba0be6d0f8601561da4e was provided as an optimization (information was taken from https://www.uclibc.org/docs/tls.pdf), and it seemed work fine. If it doesn't work for FreeBSD, we may disable this using #ifdef(s). From your explanation, I didn't completely understand, if STATIC_TLS can't work at all or only when TLS sections exceeds some limit. In first case even CLI PHP won't be able to load opcache.so. I don't have access for FreeBSD. Can you check if the patch https://gist.github.com/dstogov/c7c01283766376478b5f7073c1a432a9 fixes the problem (may be correct it, if necessary). ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=80814 -- Edit this bug report at https://bugs.php.net/bug.php?id=80814&edit=1