Bug #73265 [Opn]: Loading browscap.ini at startup causes high memory usage

Edit report at https://bugs.php.net/bug.php?id=73265&edit=1 ID: 73265 User updated by: spam2 at rhsoft dot net Reported by: spam2 at rhsoft dot net Summary: Loading browscap.ini at startup causes high memory usage Status: Open Type: Bug Package: Performance problem Operating System: Linux PHP Version: 7.0.14 Block user comment: N Private report: N New Comment: > From a quick test, performance of loading browscap.ini > did not significantly change between 5.6 and 7.0. > When you compared to PHP 5.6, were you also loading > this browscap.ini? *yes* - i synced the "php.spec" for 5.6 and 7.0 days before the upgrade so that it's drop in and only the loadmodule line in httpd needs to be changed what we did was * restart webserver * benchmark cms 1 * restart webserver * benchmark cms 2 * note numbers "dnf -y upgrade" * restart webserver * benchmark cms 1 * restart webserver * benchmark cms 2 * note numbers drop from 100 to 50 requests per second, well, that application is *using* get_browser() within a if(function_exists('get_browser')) - hence i was able to add it to 'disabled_functions' - so it's not only about loading browscap and a likely regression in 7.0.14 but also about get_browser() with the same "browscap.ini" became noticeable slower PHP 5.6.26: Requests per second: 107.24 [#/sec] (mean) Time per request: 279.741 [ms] (mean) Time per request: 9.325 [ms] (mean, across all concurrent requests) Transfer rate: 1531.03 [Kbytes/sec] received PHP 7.0.11 PGO: Requests per second: 56.30 [#/sec] (mean) Time per request: 532.864 [ms] (mean) Time per request: 17.762 [ms] (mean, across all concurrent requests) Transfer rate: 803.75 [Kbytes/sec] received PHP 7.0.14 NON-BROWSCAP: Requests per second: 327.45 [#/sec] (mean) Time per request: 61.077 [ms] (mean) Time per request: 3.054 [ms] (mean, across all concurrent requests) Transfer rate: 4754.80 [Kbytes/sec] received Previous Comments: ------------------------------------------------------------------------ [2016-12-15 13:27:36] spam2 at rhsoft dot net i have updated it on 2016-11-19 * Do Dez 08 2016 Reindl Harald <h.reindl@thelounge.net> - update to PHP 7.0.14 so i can't say it for 100% sure but i am very certain that 7.0.14 made this part much worser - what is *not* normal is zend_parse_ini_file() calling that often within a while(true) loop -> http://access.thelounge.net/harry/massif.txt since that was a cli script with a loop i would expect that only happen *once* at startup and your comment (This could also explain why you're seeing a difference in 7.0.14, because there were some changes to the ini parser) could explain some parts of it ------------------------------------------------------------------------ [2016-12-15 12:57:59] nikic@php.net Okay, not terribly surprised that loading an 8MB ini file is going to use lots of memory. From a quick test, performance of loading browscap.ini did not significantly change between 5.6 and 7.0. When you compared to PHP 5.6, were you also loading this browscap.ini? There are some optimization opportunities here: a) We shouldn't be loading the browscap.ini unless it is actually necessary. In fact, browscap.ini is already loaded lazily by get_browser() if the browscap ini option is only available during activation (but not startup). However, this mode has the disadvantage that the data is loaded per-thread and per-request. If the get_browser() function is actually used on every request, this is going to be (significantly) more expensive than loading it once. Ideally, we'd have lazy loading that still shares across requests and threads. The latter would require manual synchronization, but I think we could implement the former easily and then drop the startup loading. Threads are not relevant in most PHP deployments. b) We can optimize the loading and in-memory representation of browscap.ini. As we use a generic ini parser that is certainly not optimized for this use-case the former may be hard, but there are probably some cheap wins for the memory representation (e.g. we could intern ini keys, which for browscap are repeating). ------------------------------------------------------------------------ [2016-12-15 12:55:13] spam2 at rhsoft dot net in fact browscap is the reason for the whole bugreport, obviously with or without the memory overhead bug get_browser() which is used by the CMS of my co-developer which dropped from 100 to 50 requests per second with PHP7 luckily he is using it within function_exists() and add it to disabled_functions boosted his system from 100 requests per second with PHP5.6 to 320 with PHP 7.0.14 Requests per second: 319.89 [#/sec] (mean) Time per request: 62.522 [ms] (mean) Time per request: 3.126 [ms] (mean, across all concurrent requests) Transfer rate: 4644.90 [Kbytes/sec] received ------------------------------------------------------------------------ [2016-12-15 12:04:49] spam2 at rhsoft dot net ; [browscap] ; browscap = "/etc/php/browscap.ini" and the memory usage goes down that is the one from the servers and i think with that and my whole config in the last comment you should have a fine reproducer - will remove that from all machines now http://access.thelounge.net/harry/browscap.ini.txt ------------------------------------------------------------------------ [2016-12-15 11:41:13] spam2 at rhsoft dot net since i have one instance with a normal memory usage maybe it depends on some specific part of my php.ini and you could even reprodcue the behavior with it easily on your machine? [root@testserver:~]$ cat /etc/php.ini [PHP] default_charset = "ISO-8859-1" zend.enable_gc = 0 zend.detect_unicode = 0 register_argc_argv = 0 register_globals = 0 always_populate_raw_post_data = -1 upload_tmp_dir = "/var/www/uploadtemp" open_basedir = "" include_path = ".:/www/phpincludes:/usr/share/pear:/usr/share/php:/usr/share/php/php-reader" error_log = "/Volumes/dune/www-servers/_logs/php_error.log" zlib.output_compression = 0 zlib.output_compression_level = 4 max_execution_time = 120 max_input_time = 120 max_input_nesting_level = 32 memory_limit = "-1" post_max_size = "100M" upload_max_filesize = "100M" file_uploads = 1 max_file_uploads = 30 allow_url_fopen = 1 allow_url_include = 0 realpath_cache_size = 64K realpath_cache_ttl = 300 error_reporting = E_ALL docref_root = "http://at.php.net/manual/de/" docref_ext = ".php" disable_functions = "" disable_classes = "" engine = 1 short_open_tag = 1 asp_tags = 0 y2k_compliance = 1 output_buffering = 0 output_handler = "" implicit_flush = 0 expose_php = 0 display_errors = 1 display_startup_errors = 1 log_errors = 1 log_errors_max_len = 2048 html_errors = 0 track_errors = 0 warn_plus_overloading = 1 enable_dl = 0 cgi.force_redirect = 0 cgi.rfc2616_headers = 0 fastcgi.impersonate = 1 ignore_repeated_errors = 0 ignore_repeated_source = 0 arg_separator.output = "&amp;" arg_separator.input = "&" variables_order = "EGPCS" default_mimetype = "text/html" default_socket_timeout = 10 auto_detect_line_endings = 0 unserialize_callback_func = "" precision = 14 serialize_precision = 17 user_agent = "PHP" gd.jpeg_ignore_warning = 1 highlight.string = "#dd0000" highlight.comment = "#ff8000" highlight.keyword = "#007700" highlight.bg = "#ffffff" highlight.default = "#0000bb" highlight.html = "#000000" error_prepend_string = "" error_append_string = "" auto_prepend_file = "" auto_append_file = "" [Session] session.save_path = "/var/www/sessiondata" session.save_handler = "files" session.use_cookies = 1 session.use_only_cookies = 1 session.use_strict_mode = 1 session.name = "LOUNGE_ID" session.referer_check = "" session.auto_start = 0 session.cookie_lifetime = 0 session.cookie_path = "/" session.cookie_domain = "" session.cookie_secure = 0 session.cookie_httponly = 1 session.serialize_handler = "php" session.gc_probability = 0 session.entropy_file = "/dev/urandom" session.entropy_length = 16 session.cache_limiter = "nocache" session.cache_expire = 180 session.use_trans_sid = 0 session.bug_compat_42 = 0 session.bug_compat_warn = 0 session.hash_function = 1 session.hash_bits_per_character = 6 session.upload_progress.enabled = 1 session.lazy_write = 1 session.sid_bits_per_character = 5 session.sid_length = 40 url_rewriter.tags = "disabled" [MySQLI] mysqli.default_host = "localhost" mysqli.default_port = 3306 mysqli.default_socket = "/var/lib/mysql/mysql.sock" mysqli.default_user = "" mysqli.default_password = "" mysqli.reconnect = 1 mysqli.max_links = 300 [mysqlnd] mysqlnd.collect_statistics = 0 mysqlnd.collect_memory_statistics = 0 mysqlnd.debug = 0 mysqlnd.net_read_timeout = 60 pdo_mysql.default_socket = /var/lib/mysql/mysql.sock [Assertion] assert.active = 0 assert.quiet_eval = 0 [Sockets] sockets.use_system_read = 1 [bcmath] bcmath.scale = 20 [soap] soap.wsdl_cache_enabled = 1 soap.wsdl_cache_dir = "/var/www/uploadtemp" soap.wsdl_cache_ttl = 5 [mbstring] mbstring.language = "German" mbstring.internal_encoding = "ISO-8859-1" mbstring.http_input = "auto" mbstring.encoding_translation = 0 mbstring.detect_order = "auto" mbstring.func_overload = 0 [Date] date.timezone = "Europe/Vienna" [browscap] browscap = "/etc/php/browscap.ini" ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=73265 -- Edit this bug report at https://bugs.php.net/bug.php?id=73265&edit=1