[php-src] Issue #8360: mbstring: UTF-8 sequences with out-of-bounds follow bytes are no longer substituted correctly
| From: | alexdowad | Date: | Thu, 14 Apr 2022 06:32:32 +0000 |
| Subject: | [php-src] Issue #8360: mbstring: UTF-8 sequences with out-of-bounds follow bytes are no longer substituted correctly | ||
| Groups: | php.bugs | ||
| Request: | Send a blank email to php-bugs+get-240944@lists.php.net to get a copy of this message | ||
Issue: https://github.com/php/php-src/issues/8360
Comment Author: alexdowad
Hi, @ausi. Thanks very much for this report.
I had to search a bit to find the cause of this change and then jog my memory to remember why I made
it. (The change was in October 2020.)
The change was made in an attempt to make
mbstring's behavior when handling UTF-8
consistent with its behavior on other text encodings. For most text encodings, mbstring
emits a single error marker as soon as it sees an illegal byte, then resets itself back to its
"starting state" and continues processing the next byte.
If we take \xDF\xC0 as an example, the byte \xDF is not illegal. But then
\xC0 is. This is why I made it emit a single error marker.
However, I was not aware that there was any specific standard regarding how many errors should be
signaled for a particular *invalid* UTF-8 string. (RFC 3629 just specifies how *valid* UTF-8 strings
should be handled, not invalid ones.) In general, we want mbstring to comply as closely
as possible with all relevant standards and to be compatible with as many legacy computing systems
as possible.
Hence, I am very happy to revert this change. This time around, I would also like to 1) add some
code comments explaining why error handling for UTF-8 is done the way it is, and 2) add another test
to our test suite specifically to verify the error handling behavior for UTF-8.
To that end, I would like to ask if you have any other test cases which you can suggest should be
added to the test suite. This will help to ensure that future maintenance and ongoing development of
mbstring will not cause any further breakage.