Edit report at https://bugs.php.net/bug.php?id=81516&edit=1
ID: 81516
Comment by: roland at nextendweb dot com
Reported by: roland at nextendweb dot com
Summary: Curl used with CURLOPT_WRITEFUNCTION rarely corrupts
the response
Status: Feedback
Type: Bug
Package: cURL related
PHP Version: 7.4.24
Block user comment: N
Private report: N
New Comment:
My latest finding that the common in these servers that all of them has a PHP module called i360
which is a security module developed by https://www.imunify360.com/
It seems like that their module identifies that given chunk as a threat. I get in touch with the
developer of that module and I will update this thread...
Previous Comments:
------------------------------------------------------------------------
[2021-10-13 23:19:39] requinix@php.net
Weird. What happens if you have PHP output the data instead and then pipe that into cat? Or use some
other methods of writing out to a file?
<?php
$value = '...';
echo substr(base64_decode($value), 0, 1100);
?>
$ php nowrite.php | cat > b.txt
------------------------------------------------------------------------
[2021-10-13 17:05:58] roland at nextendweb dot com
It looks like these are some kind of special systems which are unable to write the following data to
the filesystem and this is why that chunk is missing always.
https://gist.github.com/nextend/91b09c70a5a86fff34bcf87e29c9f068
Output is NULL and the file is not created in the filesystem. No error thrown.
------------------------------------------------------------------------
[2021-10-13 16:49:03] roland at nextendweb dot com
https://gist.github.com/nextend/8c0d752024bac9c0a030d5ba022dfac9
There is a chunk which is not written into the stream. The output is the following:
/ 1160
Chunk #4430: 781473d903bedf95d8a07d34974ac548 1160
Good md5: 231732259d67fe83ed6fc02d7ad9be57
Stream md5: 4a3b968a44c585a2883e687d61c251fb
Memory md5: 231732259d67fe83ed6fc02d7ad9be57
The return value of fwrite() for this chunk is NULL.
------------------------------------------------------------------------
[2021-10-13 16:35:59] roland at nextendweb dot com
The following gist works like a charm and the md5 of the data is fine without the PHP stream write:
https://gist.github.com/nextend/61bb5a8b36ea126f3e95297f3f7de10e
I tried, but it takes forever to finish:
for ($len = 0; $len < $data_length; ) {
$len += fwrite($this->stream_handle, substr($data, $len));
}
------------------------------------------------------------------------
[2021-10-13 08:15:11] roland at nextendweb dot com
Thank you for your tips!
As you wrote CURLOPT_BUFFERSIZE is just a request and not an order, so it might happen that a part
of the response is missing and this is why the md5 hashes are different for several chunks. So I can
see why my check of the chunk's md5 hashes might not the right tool for debugging this.
Expected response at buffersize 4:
aaaa
bbbb
cccc
dddd
-> response hash OK
Valid response at buffersize 4:
aaaa
bbb -> chunk hash mismatch
bcccc -> chunk hash mismatch
dddd
-> response hash OK
Not valid response at buffersize 4:
aaaa
bb -> chunk hash mismatch
ccc -> chunk hash mismatch
dddd
-> response hash FAIL
I think the latter happens, but I will do more debugging with the length of the chunks.
Currently I do not have access to the server which produced this error, but will try to get one. We
know about 10 different servers which produced this and they are were the very same response MD5
hash, so it could not be a coincident.
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=81516
--
Edit this bug report at https://bugs.php.net/bug.php?id=81516&edit=1