RE: [PHP-DEV] readfile() memory usage

Hi, Readfile() is internally implemented in the same way like fpassthru() (actually the same backend function is called, readfile only opening a stream before delegating to passthru). Both methods delegate from PHP user-space to an internal streams API methods php_stream_passthru(). This one has 2 implementations: - If the underlying stream allows MMAP, it will use memory mapping (mapping file to *virtual* memory) and copy the mapped buffer to output. Please note memory mapping does *not* load the file into memory, it only *maps* the file contents to virtual memory like a swap file (http://en.wikipedia.org/wiki/Mmap). - If this is not the case, it copies the whole file in blocks of 8192 bytes using a conventional loop. I verified, this code is at least in PHP 5.2 and 5.3, maybe earlier, too. Uwe ----- Uwe Schindler thetaphi@php.net - http://www.php.net NSAPI SAPI developer Bremen, Germany > -----Original Message----- > From: Larry Garfield [mailto:larry@garfieldtech.com] > Sent: Monday, April 30, 2012 7:22 AM > To: internals@lists.php.net > Subject: [PHP-DEV] readfile() memory usage > > So, I've been reading articles for a decade now that say that readfile() is great > and wonderful except for memory usage. Specifically, that it reads a file into > memory entirely, and then prints it to stdout from there. So if you're outputing > a big file you will hit your memory limit and kill the server. Thus, one should > always loop over fread() instead. The most recent article I found saying that > was from 2007, with a StackExchange thread saying the same from 2011. I've > even found mention of it in old PHP Bugs. > > However, I cannot replicate that in my own testing. Earlier today I was running > some benchmarks of different file streaming techniques in PHP > (5.3.6 specifically) and found that fread() looping, fpassthru(), readfile(), and > stream_copy_to_stream() perform almost identically on memory, and all are > identical on CPU except for fread() which is slower, which makes sense since > you're looping in PHP space. > > What's more, I cranked my memory limit down to 10 MB and then tried > streaming a 20 MB file. No change. The PHP peak memory never left around > a half-meg or so, most of which I presume is just the Apache/PHP overhead. > But it's not actually possible for readfile() to be buffering the whole file into > memory before printing and not die if the file is bigger than the memory limit. > I verified that the data I'm getting downloaded from the script is correct, and > exactly matches the file that it should be streaming. > > My first thought was that this is yet another case of PHP improving and fixing a > long-standing bug, but somehow the rest of the world not knowing about it so > "conventional wisdom" persists long after it's still wise. However, I found no > mention of readfile() in the PHP 5 change log[1] at all aside from one note > from back in 5.0.0 Beta 1 about improving performance under Windows. (I'm > on Linux.) > > So, what's going on here? Has readfile() been memory-safe for that long > without anyone noticing? Is my test completely flawed (although I don't see > how since I can verify that the code works as expected)? Something else? > > Please un-confuse me! > > (Note: Sending this to internals since this is an engine question, and I am more > likely to reach whoever it was that un-sucked readfile() sometime in the silent > past that way. <g>) > > --Larry Garfield > > -- > PHP Internals - PHP Runtime Development Mailing List To unsubscribe, visit: > http://www.php.net/unsub.php