#29922 [Asn]: Huge memory usage in file copy of big files

ID: 29922 Updated by: wez@php.net Reported By: info at pion dot xs4all dot nl Status: Assigned Bug Type: *Directory/Filesystem functions Operating System: Linux PHP Version: 5.0.1 Assigned To: wez New Comment: IIRC, I added a limit for this kind of thing for readfile() (aka php_stream_passthru), which could be applied here. Discussions on the list (with Andi) suggested mmap'ing chunks of 1MB and doing it that way. If someone wants to cook up a patch for either of these, that's fine by me; I'm not sure how soon I'll be able to look into it. Previous Comments: ------------------------------------------------------------------------ [2004-09-02 08:19:24] derick@php.net I saw a similar (but better) patch by Wez. Wez, can you have a look? ------------------------------------------------------------------------ [2004-09-01 22:19:57] info at pion dot xs4all dot nl I have a small patch for the 5.1.0-dev which solves the problem but: This patch says a normal copy should be used for files bigger then 32 MB. But this is an rather arbitrary value choosen by me. Here follows the patch: -diff -Naur org/php5-200409011830/main/streams/streams.c php5-200409011830/main/streams/streams.c --- org/php5-200409011830/main/streams/streams.c 2004-08-31 14:30:59.000000000 +0200 +++ php5-200409011830/main/streams/streams.c 2004-09-01 22:06:11.579747624 +0200 @@ -1276,7 +1276,12 @@ } } - if (php_stream_mmap_possible(src)) { + /* + ** We shouldn't use memory mapping on big files just for copying them. + ** It's not an efficient sollution for the vm. So for files above + ** 32mb (32*1024^2=33554432) we want a normal copy + */ + if ((php_stream_mmap_possible(src)) && (ssbuf.sb.st_size < 33554432)) { char *p; size_t mapped; ---------------------- ------------------------------------------------------------------------ [2004-09-01 21:55:01] info at pion dot xs4all dot nl I have checked again but i can't find the bug report. I have tested with "5.1.0-dev" with checked out version 200409011830 but the problem is still there. ------------------------------------------------------------------------ [2004-09-01 08:44:03] derick@php.net I'm sure there is already a bugreport for this, and afaik somebody also worked on an attempt to fix it. Can you try a PHP 5.1.0-dev snapshot please? (http://naps.php.net) ------------------------------------------------------------------------ [2004-08-31 22:03:04] info at pion dot xs4all dot nl - The observation was on rhel 3 with php 4.3.2 - The test in the report was done on a FC1 system with php 4.3.3 - Now i have checked php 4.3.8 on fc2 (the default php) and php 5.0.1 Al have the same result although the speed diferences on the fc2 system are smaller. This could be caused by different hardware or differences in the linux vm (2.4.X versus 2.6.X) ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at http://bugs.php.net/29922 -- Edit this bug report at http://bugs.php.net/?id=29922&edit=1