Edit report at https://bugs.php.net/bug.php?id=81516&edit=1
ID: 81516
Updated by: requinix@php.net
Reported by: roland at nextendweb dot com
Summary: Curl used with CURLOPT_WRITEFUNCTION rarely corrupts
the response
-Status: Open
+Status: Feedback
Type: Bug
Package: cURL related
PHP Version: 7.4.24
Block user comment: N
Private report: N
Previous Comments:
------------------------------------------------------------------------
[2021-10-12 18:49:18] requinix@php.net
Can you modify your script to also include the data length in the log output?
After doing that and running the script (and if seeing the length doesn't identify the
problem), can you additionally modify the fwrite() statement to be a loop:
for ($len = 0; $len < $data_length; ) {
$len += fwrite($this->stream_handle, substr($data, $len));
}
This is because fwrite() does not guarantee to write its entire string at once, and will return the
number of bytes it actually wrote. This is almost never an issue with writing to a local file,
though...
Back to the data length. The output shows those 40 chunks *present* but with different hashes. Your
original report said that 1160 bytes were *missing*. Does that mean you're seeing two different
behaviors between the WP code and your script?
Because if cURL is giving the write function 1160 bytes, that means your original report had a
problem with the callback 1 time while your new script had a problem with the callback 40 times.
Speaking of, note that the buffer size is "a request, not an order", so the chunks
received by the callback may not be exactly that size.
https://curl.se/libcurl/c/CURLOPT_BUFFERSIZE.html
------------------------------------------------------------------------
[2021-10-12 13:51:27] roland at nextendweb dot com
Also worth to mention that this issue does not happen when the server which serves the zip has
HTTP/2 enabled. It happened only with HTTP/1.1
------------------------------------------------------------------------
[2021-10-12 12:57:56] roland at nextendweb dot com
I was able to create a small test case which can reproduce the issue without WordPress.
phpinfo(); -> https://gist.github.com/nextend/3905674a48c2da5c95dfea90b2115f82
PHP 7.4.24
cURL 7.71.0
Test code:
https://gist.github.com/nextend/2819c93635c5f1c68c13cf1e28a2ef19
The problem happens at CURLOPT_BUFFERSIZE => 1160, 1160*2, 1160*3
1160*4 seems to work fine.
See the md5 hash of the stream write data at 1160 buffer size. (Left is the normal server, right is
the bad server) https://i.imgur.com/Jk0hzaM.png
Between 3683 and 3711 the chunk's md5 are mismatching.
If you want to I can send you the private url for the test in email.
------------------------------------------------------------------------
[2021-10-08 18:38:44] requinix@php.net
Are you sure this is related to CURLOPT_WRITEFUNCTION? Can you confirm that stream_body is not being
called for some chunk in the middle of the response? Would be nice if you could modify cURL.php to
not need the write function, but with a casual glance at the code I don't think it would be
easy.
How about varying the buffer size to some fraction/multiple of 1160, like 580 or 2320?
And, of course, are there any relevant differences in configuration between the servers? How about
with the files or paths being written to?
No offense but this doesn't really sound like a bug in PHP, or even in cURL itself, but rather
some weird hiccup or fault with those specific servers.
------------------------------------------------------------------------
[2021-10-08 17:52:08] roland at nextendweb dot com
Description:
------------
This issue happened through WordPress update system during plugin update. Our update url points to
our private server and serving a zip file. We had tens of thousands updates for this very same file
and there were only 2 sites which had this issue, so I think it is not related how we serve the
file.
The error is md5_mismatch: The checksum of the file (4a3b968a44c585a2883e687d61c251fb) does not
match the expected checksum value (231732259d67fe83ed6fc02d7ad9be57).
I started to debug this issue, I had the original zip file and the corrupted one which WordPress
downloaded. I diffed the files and at some point it seems like some part is missing in the corrupted
file. Like the stream lost a piece from the buffer. And it happens all the time on those servers.
https://i.imgur.com/Nx0SKeK.png
Curl transport handled this download, so I head over to /wp-includes/Requests/Transport/cURL.php: https://github.com/WordPress/WordPress/blob/master/wp-includes/Requests/Transport/cURL.php
And when I removed CURLOPT_BUFFERSIZE or changed anything else (Requests::BUFFER_SIZE+1,
Requests::BUFFER_SIZE-1), download_url function started to behave as it should and the file was not
corrupted anymore.
<?php
curl_setopt($this->handle, CURLOPT_BUFFERSIZE, Requests::BUFFER_SIZE);
Requests::BUFFER_SIZE value is 1160 in WordPress.
Exactly one 1160 bytes block is missing in the middle of the file, which is the value of
Requests::BUFFER_SIZE.
https://i.imgur.com/KoWkzco.png
Both site had the following configuration:
WordPress 5.8.1
PHP 7.4.24
Curl 7.71.0
------------------------------------------------------------------------
--
Edit this bug report at https://bugs.php.net/bug.php?id=81516&edit=1