Bug #60274 [Asn->Fbk]: cpu steady on httpd processes never clears

Edit report at https://bugs.php.net/bug.php?id=60274&edit=1 ID: 60274 Updated by: sixd@php.net Reported by: jwprice at georgiasouthern dot edu Summary: cpu steady on httpd processes never clears -Status: Assigned +Status: Feedback Type: Bug Package: OCI8 related Operating System: Solaris 10 PHP Version: 5.3.8 Assigned To: sixd Block user comment: N Private report: N New Comment: The MaxRequestsPerChild setting will cause the httpd processes to restart after a number of HTTP requests. Your MaxClients looks inherently high; but only your application tuning can determine the best value. It effectively means you could have up to 1200 connections open to the DB, which will cause problems in the DB tier. I suggest: - Do the PHP & Oracle client upgrade previously discussed. - Bring the OS patch level up to date. - Review the application for inefficiencies - fixing these should help reduce load in each tier. - Check for application errors and for Oracle trace files. using Oracle Instant Client 11g will create better client side trace files. - Monitor if anything unexpected is causing load during "idle" times. - Check that the number of httpd processes decreases when there is no load. - Use truss, pstack etc to identify what any high CPU httpd process are doing Previous Comments: ------------------------------------------------------------------------ [2012-02-29 13:42:44] jwprice at georgiasouthern dot edu What do you mean by httpd process cycled? Do you be restarted every so often? I was not aware there were and php.ini oci settings so we're using whatever the defaults are. The database is on a separate machine. These settings have been set for a couple of years now. We changed MaxClients to 1200 an it solved couple of problems we were having and worked for one fall registration. We only started having a problem when we upgraded php and apache from php 5.2.11 and apache 2.0.63 <IfModule prefork.c> ServerLimit 1200 StartServers 5 MinSpareServers 5 MaxSpareServers 10 MaxClients 1200 MaxRequestsPerChild 5000 </IfModule> ------------------------------------------------------------------------ [2012-02-29 00:09:59] sixd@php.net Are the httpd processes not being cycled by choice? What are the MaxClients and MaxRequestsPerChild values? (Are you using the prefork MPM?). What are the php.ini oci8.* values? Is the DB on a different host? oci_pconnect will cause DB host memory contention unless you restrict MaXClients and/or have Oracle 11g and use its DRCP. ------------------------------------------------------------------------ [2012-02-28 19:46:39] jwprice at georgiasouthern dot edu cpu load was measured with uptime and prstat. Note this is a 4 core 16 thread 16Gb SunFire T2000. Load is normally between 3 and 5, between 350 and 550 process. During the busy time load was > 200 and stayed there even after the load on the database was gone. I would say that the httpd processes were spinning at 3 to 4% of cpu each normally this is a lot lower .1%. They are not being cycled. We are using an older version of the oracle client we will definitely try to update it. We have since updated to apache 2.2.21 and php- 5.3.8. We will have another high load Mar 5th. We've stayed away from persistent connections because it caused a high load on the database. We will look into the performance tips you mentioned. Apache was configured and compiled with the following: ./configure --enable-modules=most \ --prefix=${Home}/server \ --enable-mods-shared=most \ --enable-proxy \ --enable-proxy-http \ --enable-so ------------------------------------------------------------------------ [2012-02-28 18:39:15] sixd@php.net This bug just came to my attention and I've now changed the package from "Performance" to "OCI8". How are you measuring CPU load? How is Apache configured and what is the system configuration? Are httpd process spinning? Are they being cycled? 600 processes seems a large number for a single commodity machine. One old rule of thumb was to set maxclients to 25 for a 1Gb, 2-core machine. Look at reducing maxclients and using persistent OCI8 connections. If possible, use Oracle 11g DRCP connections. I'd recommend upgrading the Oracle client by using the free Oracle Instant Client. Use the latest version that can still connect back to your DB. The current PHP OCI8 hasn't had any particularly relevant changes since 5.3.4. Make sure the code uses binds, prefetches, etc. effectively. Check the performance tips in http://www.oracle.com/technetwork/topics/php/underground-php-oracle-manual- 098250.html ------------------------------------------------------------------------ [2011-11-11 21:21:27] jwprice at georgiasouthern dot edu Description: ------------ During a busy time over 600 httpd processes on the webserver and oracle database cpu load over 88. The cpu load on each httpd process climbs to 3 and 4 % over loading our server forcing cpuloads of > 200 on the webserver. After load on the database clears usually in about 15 min, the httpd process never clear their cpu usage even after a week of no activity. The cpu load on the httpd processes usally ranges from .4 to 0%. The cpu load problem on the webserver is definitely related to slowness on the oracle database side, but what concerns me is the apache server never clears itself. I suspect there is some issue with the oci functionality. I'm reporting this before upgrading to the latest version because i don't see any fixes related. We are using php version 5.3.4. Configure script below. Home=/portal ApacheStdName=httpd- ApacheVer=2.2.17 PHPVer=5.3.4 PHPStdName=php- OracleClientHome=/oraclient/ora920 MysqlHome=/usr/local/mysql ModSSLStdName=mod_ssl- ModSSLVer=2.8.22-1.3.33 cd ${Home}/${PHPStdName}${PHPVer} ./configure \ --prefix=${Home}/local \ --with-apxs2=${Home}/server/bin/apxs \ --with-gettext \ --enable-ftp \ --with-openssl=/usr/local/ssl \ --with-mysql=${MysqlHome} \ --with-pear=${HOME}/pear \ --enable-cli \ --disable-cgi \ --with-curl \ --with-mcrypt=/usr/local \ --with-zlib \ --with-zlib-dir=/usr/local \ --with-jpeg-dir=/usr/local \ --with-png-dir=/usr/local \ --with-xpm-dir=/usr/local \ --with-freetype-dir=/usr/local \ --enable-gd-native-ttf \ --with-ttf=/usr/local \ --with-gd \ --with-ldap=/usr/local \ --with-oci8=${OracleClientHome} \ --with-config-file-path=${Home}/etc Test script: --------------- We use oci_connect ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=60274&edit=1