Bug #81516 [Com]: Curl used with CURLOPT_WRITEFUNCTION rarely corrupts the response

Edit report at https://bugs.php.net/bug.php?id=81516&edit=1 ID: 81516 Comment by: roland at nextendweb dot com Reported by: roland at nextendweb dot com Summary: Curl used with CURLOPT_WRITEFUNCTION rarely corrupts the response Status: Feedback Type: Bug Package: cURL related PHP Version: 7.4.24 Block user comment: N Private report: N New Comment: The following gist works like a charm and the md5 of the data is fine without the PHP stream write: https://gist.github.com/nextend/61bb5a8b36ea126f3e95297f3f7de10e I tried, but it takes forever to finish: for ($len = 0; $len < $data_length; ) { $len += fwrite($this->stream_handle, substr($data, $len)); } Previous Comments: ------------------------------------------------------------------------ [2021-10-13 08:15:11] roland at nextendweb dot com Thank you for your tips! As you wrote CURLOPT_BUFFERSIZE is just a request and not an order, so it might happen that a part of the response is missing and this is why the md5 hashes are different for several chunks. So I can see why my check of the chunk's md5 hashes might not the right tool for debugging this. Expected response at buffersize 4: aaaa bbbb cccc dddd -> response hash OK Valid response at buffersize 4: aaaa bbb -> chunk hash mismatch bcccc -> chunk hash mismatch dddd -> response hash OK Not valid response at buffersize 4: aaaa bb -> chunk hash mismatch ccc -> chunk hash mismatch dddd -> response hash FAIL I think the latter happens, but I will do more debugging with the length of the chunks. Currently I do not have access to the server which produced this error, but will try to get one. We know about 10 different servers which produced this and they are were the very same response MD5 hash, so it could not be a coincident. ------------------------------------------------------------------------ [2021-10-12 18:49:18] requinix@php.net Can you modify your script to also include the data length in the log output? After doing that and running the script (and if seeing the length doesn't identify the problem), can you additionally modify the fwrite() statement to be a loop: for ($len = 0; $len < $data_length; ) { $len += fwrite($this->stream_handle, substr($data, $len)); } This is because fwrite() does not guarantee to write its entire string at once, and will return the number of bytes it actually wrote. This is almost never an issue with writing to a local file, though... Back to the data length. The output shows those 40 chunks *present* but with different hashes. Your original report said that 1160 bytes were *missing*. Does that mean you're seeing two different behaviors between the WP code and your script? Because if cURL is giving the write function 1160 bytes, that means your original report had a problem with the callback 1 time while your new script had a problem with the callback 40 times. Speaking of, note that the buffer size is "a request, not an order", so the chunks received by the callback may not be exactly that size. https://curl.se/libcurl/c/CURLOPT_BUFFERSIZE.html ------------------------------------------------------------------------ [2021-10-12 13:51:27] roland at nextendweb dot com Also worth to mention that this issue does not happen when the server which serves the zip has HTTP/2 enabled. It happened only with HTTP/1.1 ------------------------------------------------------------------------ [2021-10-12 12:57:56] roland at nextendweb dot com I was able to create a small test case which can reproduce the issue without WordPress. phpinfo(); -> https://gist.github.com/nextend/3905674a48c2da5c95dfea90b2115f82 PHP 7.4.24 cURL 7.71.0 Test code: https://gist.github.com/nextend/2819c93635c5f1c68c13cf1e28a2ef19 The problem happens at CURLOPT_BUFFERSIZE => 1160, 1160*2, 1160*3 1160*4 seems to work fine. See the md5 hash of the stream write data at 1160 buffer size. (Left is the normal server, right is the bad server) https://i.imgur.com/Jk0hzaM.png Between 3683 and 3711 the chunk's md5 are mismatching. If you want to I can send you the private url for the test in email. ------------------------------------------------------------------------ [2021-10-08 18:38:44] requinix@php.net Are you sure this is related to CURLOPT_WRITEFUNCTION? Can you confirm that stream_body is not being called for some chunk in the middle of the response? Would be nice if you could modify cURL.php to not need the write function, but with a casual glance at the code I don't think it would be easy. How about varying the buffer size to some fraction/multiple of 1160, like 580 or 2320? And, of course, are there any relevant differences in configuration between the servers? How about with the files or paths being written to? No offense but this doesn't really sound like a bug in PHP, or even in cURL itself, but rather some weird hiccup or fault with those specific servers. ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=81516 -- Edit this bug report at https://bugs.php.net/bug.php?id=81516&edit=1