Random Performance Problems

We launched a new PHP application on our site last week after months of development/testing and have been observing some random performance problems that I'm hoping someone might be able to provide insight into. We're extracting information from Oracle for around 400 unique programs of study and we're currently getting around 7-8,000 page views per day. We are averaging around 1.29 seconds per page, which isn't bad, but 3% of our page views take over 5 seconds and 1% are taking over 10 seconds - some way over that. One would think that it would have something to do with the varying complexity of pages, but this only partially explains it. We have lots of pages where the average is ~2 seconds but they get an occasional extremely long time like 178 seconds, 154 seconds, 60 seconds etc. It doesn't have anything to do with load on the server or database as consecutive page views on either side of these extremely long ones perform great and these random delays are scattered evenly throughout a 24 hour period. The application is extracting data from an Oracle db, saving the data in an extremely multidimensional array (10 dimensions deep in some places) and then laying out the extracted data in HTML. We've added getmicrotime() code to the very top of the page and the very bottom, as well as selected places within the code to try and narrow down the performance problem cause. These numbers are logged to a file at the very end of processing. Very infrequently the time in the db extraction portion seems to be the bottleneck, while most of the time it seems to delay randomly somewhere in the layout portion -- which isn't doing anything special other than: while (list ($key, $data) = each ($data_array)) { ... } loops (we eliminated the foreach loops thinking that this is where the problem was). I've created two PDF reports which show the numbers we are getting: http://programs.bcit.ca/pdf/views.pdf [56KB] http://programs.bcit.ca/pdf/scatter.pdf [104KB] The first lists all of the pages with average and minimum times, sorted by maximum generation time while the second plots each page generation time over a 24 hour period. The application is at http://programs.bcit.ca/ if anyone is interested and the current environment is Apache 1.3.23 with PHP 4.1.2 hitting an Oracle 8i db. If I had to take a guess at what was causing this, I would say that garbage collection was occurring at random times, but on searching for more info on this with PHP, I couldn't find anything conclusive. Any help or insight is appreciated!