Bug #73262 [Opn]: 8192 CHUNK_SIZE and PHP_SOCK_CHUNK_SIZE makes copy() slow

Edit report at https://bugs.php.net/bug.php?id=73262&edit=1 ID: 73262 User updated by: jeffreydwalter at gmail dot com Reported by: jeffreydwalter at gmail dot com Summary: 8192 CHUNK_SIZE and PHP_SOCK_CHUNK_SIZE makes copy() slow Status: Open Type: Bug Package: Performance problem Operating System: Irrelevant PHP Version: Irrelevant Block user comment: N Private report: N New Comment: I also forgot to mention that I tested this in PHP 5.6.20 on FreeBSD 10.3, but it appears to be an issue across all versions of PHP, including 7.x. Previous Comments: ------------------------------------------------------------------------ [2016-10-07 01:13:59] jeffreydwalter at gmail dot com After further testing it seems like setting PHP_SOCK_CHUNK_SIZE = 8192*24 is enough to speed up copy(), which make sense I suppose since it's using a socket to do the writes. ------------------------------------------------------------------------ [2016-10-07 01:05:46] jeffreydwalter at gmail dot com Description: ------------ ***I have not tested this outside of FreeBSD 10.3, but it should be an issue on most if not all platforms*** copy() is several orders of magnitude slower than exec('cp ...') on the same partition. It is even worse copying across partitions. After digging through the source code, I discovered that it's because of the shear number of buffered write() calls that are made on the underlying socket in _php_stream_write_buffer(). The buffer's size is hardcoded to 8192 <a href="https://github.com/php/php-src/blob/PHP-5.6.20/main/streams/php_streams_int.h#L49">here</a> and <a href="https://github.com/php/php-src/blob/PHP-5.6.20/main/php_network.h#L210">here</a>. If I change CHUNK_SIZE and PHP_SOCK_CHUNK_SIZE to 8192*24, the php copy() function will perform at a comparable speed to exec('cp ...'). I don't fully understand the ramifications of changing that buffer outside of greater memory usage, and I haven't tested it on network-related stream operations. So if there's a better way I'd love to know. I do realize that I could create a stream context in php, set the buffer size, and pass that context to copy(), but that doesn't seem like a workable solution. Perhaps a better solution would be to make those CHUNK_SIZE defines tunable at compile time so they can be optimized for one's specific platform? Test script: --------------- <?php # Change this is you want to test with a different file size. define('FILE_SIZE_IN_MB', 250); printf('Generating %sMB dummy file.'.PHP_EOL, FILE_SIZE_IN_MB); echo shell_exec('dd if=/dev/urandom of=dummy.txt bs=1048576 count='.FILE_SIZE_IN_MB.' 2>&1 | grep -v records'); $copy_start = time(); copy('dummy.txt', 'copied_with_php_copy.txt'); $copy_done = time(); $cp_start = time(); exec('cp dummy.txt copied_with_cp.txt'); $cp_done = time(); echo 'copy() took: '.($copy_done-$copy_start).'s'.PHP_EOL; echo 'cp took: '.($cp_done-$cp_start).'s'.PHP_EOL; echo shell_exec('ls -al dummy.txt copied_with_php_copy.txt copied_with_cp.txt').PHP_EOL; echo 'Removing test files.'.PHP_EOL; exec('rm -rf dummy.txt copied_with_php_copy.txt copied_with_cp.txt'); Expected result: ---------------- php_stream_write took 0 seconds 796 milliseconds (I added this line in stream.c) php_stream_write() with mmap used (I added this line in stream.c) php output: Generating 250MB dummy file. copy() took: 1s cp took: 1s -rw-r--r-- 1 262144000 Oct 6 20:00 copied_with_cp.txt -rw-r--r-- 1 262144000 Oct 6 20:00 copied_with_php_copy.txt -rw-r--r-- 1 262144000 Oct 6 19:59 dummy.txt Removing test files. Actual result: -------------- php_stream_write took 5 seconds 562 milliseconds (I added this line in stream.c) php_stream_write() with mmap used (I added this line in stream.c) php output: php blah.php Generating 250MB dummy file. copy() took: 13s cp took: 0s -rw-r--r-- 1 262144000 Oct 6 20:00 copied_with_cp.txt -rw-r--r-- 1 262144000 Oct 6 20:00 copied_with_php_copy.txt -rw-r--r-- 1 262144000 Oct 6 19:59 dummy.txt Removing test files. ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=73262&edit=1