Tale of woe and intrigue

This message was sent from Geocrawler.com by "Brad Jackson" <brad_jackson@inlet.com> Be sure to reply to that address. Hardware: Dual PII 450, 512MB RAM, 1GB swap Software: RedHat Linux 6.0, stock 2.2.5 kernel, PHP 3.0.12 Recently, I wrote a utility that retrieves XML files from an archive on another server and parses them to make sure there are no errors. I was surprised to see that some of the files were over 200 MB (uncompressed) and one was 475 MB! I figured with 1 GB of swap, it could handle it with no problem. However, the server started to lock up frequently and stop responding to everything except pinging and the machine had to be reset. I tried this same procedure of reading large files on another box that is running RedHat 6.1 and the problem wasn't there. It would read 500-800 MB files with no lockup. Another server running RedHat 6.0 died the same way. We decided to upgrade the kernel to 2.2.14 on the second test machine and the problem went away. After upgrading the original server to 2.2.14, the problem is gone there too. So it appears to be a kernel bug. Now, the reason I'm writing this is that during this testing, I learned about how PHP allocates memory after the file is read. We were running "top d0" and watching the memory usage of the httpd as it read a multi-hundred megabyte file from disk into a string variable with fopen/fread. After the entire file is read, the memory usage appears to double as if strcpy or memcpy is being called on the buffer. So reading a 200 MB file will cause the httpd to use 400 MB of RAM for a brief instance, or not so brief if it has to go into swap hundreds of megabytes. So, I'd like to request that someone take a look at the logic of this code and see if this copying of the buffer that doubles memory use can be eliminated. I'd think you would be able to copy a pointer and not the entire buffer. Below is code similar to what I was using to test. <?php // Run "top d0" and press Shift+M to sort by memory usage set_time_limit(0); $tmpFileName = "/tmp/4018105"; // 700 MB file (Invoice no is 4018105) $fp = fopen($tmpFileName, "rb"); echo "reading $tmpFileName<br>"; flush(); $content = fread($fp, filesize($tmpFileName)); fclose($fp); // Memory use is over double the file size at this point echo "Sleeping 10 seconds<br>"; flush(); sleep(10); echo "Done"; ?> Geocrawler.com - The Knowledge Archive