Bug #73262 [Opn]: 8192 CHUNK_SIZE and PHP_SOCK_CHUNK_SIZE makes copy() slow

Edit report at https://bugs.php.net/bug.php?id=73262&edit=1 ID: 73262 Updated by: ab@php.net Reported by: jeffreydwalter at gmail dot com Summary: 8192 CHUNK_SIZE and PHP_SOCK_CHUNK_SIZE makes copy() slow Status: Open Type: Bug Package: Performance problem Operating System: Irrelevant PHP Version: Irrelevant Block user comment: N Private report: N New Comment: Thanks for the report. You seem to be on a machine with quite a slow I/O. While it is true, that increasing the buffer will bring a speedup, in some cases it will be on cost of wasted memory. Also, there are likely situations where PHP is even faster. Have you checked various filesizes? OFC it is very platform and HW dependent. It could be indeed an idea, to set the buffer size through the context on runtime. Another option could be to extend the buffer automatically. Hardcoding the buffer size at compilation time is IMHO not that sensible. Different streams will possibly need different handling. Fe for a stream that only handles a few bytes of data, having a buffer of 192k as you do is for sure an overkill. So a balanced solution could be challenging. Thanks. Previous Comments: ------------------------------------------------------------------------ [2016-10-07 01:15:14] jeffreydwalter at gmail dot com I also forgot to mention that I tested this in PHP 5.6.20 on FreeBSD 10.3, but it appears to be an issue across all versions of PHP, including 7.x. ------------------------------------------------------------------------ [2016-10-07 01:13:59] jeffreydwalter at gmail dot com After further testing it seems like setting PHP_SOCK_CHUNK_SIZE = 8192*24 is enough to speed up copy(), which make sense I suppose since it's using a socket to do the writes. ------------------------------------------------------------------------ [2016-10-07 01:05:46] jeffreydwalter at gmail dot com Description: ------------ ***I have not tested this outside of FreeBSD 10.3, but it should be an issue on most if not all platforms*** copy() is several orders of magnitude slower than exec('cp ...') on the same partition. It is even worse copying across partitions. After digging through the source code, I discovered that it's because of the shear number of buffered write() calls that are made on the underlying socket in _php_stream_write_buffer(). The buffer's size is hardcoded to 8192 <a href="https://github.com/php/php-src/blob/PHP-5.6.20/main/streams/php_streams_int.h#L49">here</a> and <a href="https://github.com/php/php-src/blob/PHP-5.6.20/main/php_network.h#L210">here</a>. If I change CHUNK_SIZE and PHP_SOCK_CHUNK_SIZE to 8192*24, the php copy() function will perform at a comparable speed to exec('cp ...'). I don't fully understand the ramifications of changing that buffer outside of greater memory usage, and I haven't tested it on network-related stream operations. So if there's a better way I'd love to know. I do realize that I could create a stream context in php, set the buffer size, and pass that context to copy(), but that doesn't seem like a workable solution. Perhaps a better solution would be to make those CHUNK_SIZE defines tunable at compile time so they can be optimized for one's specific platform? Test script: --------------- <?php # Change this is you want to test with a different file size. define('FILE_SIZE_IN_MB', 250); printf('Generating %sMB dummy file.'.PHP_EOL, FILE_SIZE_IN_MB); echo shell_exec('dd if=/dev/urandom of=dummy.txt bs=1048576 count='.FILE_SIZE_IN_MB.' 2>&1 | grep -v records'); $copy_start = time(); copy('dummy.txt', 'copied_with_php_copy.txt'); $copy_done = time(); $cp_start = time(); exec('cp dummy.txt copied_with_cp.txt'); $cp_done = time(); echo 'copy() took: '.($copy_done-$copy_start).'s'.PHP_EOL; echo 'cp took: '.($cp_done-$cp_start).'s'.PHP_EOL; echo shell_exec('ls -al dummy.txt copied_with_php_copy.txt copied_with_cp.txt').PHP_EOL; echo 'Removing test files.'.PHP_EOL; exec('rm -rf dummy.txt copied_with_php_copy.txt copied_with_cp.txt'); Expected result: ---------------- php_stream_write took 0 seconds 796 milliseconds (I added this line in stream.c) php_stream_write() with mmap used (I added this line in stream.c) php output: Generating 250MB dummy file. copy() took: 1s cp took: 1s -rw-r--r-- 1 262144000 Oct 6 20:00 copied_with_cp.txt -rw-r--r-- 1 262144000 Oct 6 20:00 copied_with_php_copy.txt -rw-r--r-- 1 262144000 Oct 6 19:59 dummy.txt Removing test files. Actual result: -------------- php_stream_write took 5 seconds 562 milliseconds (I added this line in stream.c) php_stream_write() with mmap used (I added this line in stream.c) php output: php blah.php Generating 250MB dummy file. copy() took: 13s cp took: 0s -rw-r--r-- 1 262144000 Oct 6 20:00 copied_with_cp.txt -rw-r--r-- 1 262144000 Oct 6 20:00 copied_with_php_copy.txt -rw-r--r-- 1 262144000 Oct 6 19:59 dummy.txt Removing test files. ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=73262&edit=1