[php-src] Issue #8360: mbstring: UTF-8 sequences with out-of-bounds follow bytes are no longer substituted correctly

From: Date: Thu, 14 Apr 2022 06:32:32 +0000
Subject: [php-src] Issue #8360: mbstring: UTF-8 sequences with out-of-bounds follow bytes are no longer substituted correctly
Groups: php.bugs 
Request: Send a blank email to php-bugs+get-240944@lists.php.net to get a copy of this message
Issue: https://github.com/php/php-src/issues/8360 Comment Author: alexdowad Hi, @ausi. Thanks very much for this report. I had to search a bit to find the cause of this change and then jog my memory to remember why I made it. (The change was in October 2020.) The change was made in an attempt to make mbstring's behavior when handling UTF-8 consistent with its behavior on other text encodings. For most text encodings, mbstring emits a single error marker as soon as it sees an illegal byte, then resets itself back to its "starting state" and continues processing the next byte. If we take \xDF\xC0 as an example, the byte \xDF is not illegal. But then \xC0 is. This is why I made it emit a single error marker. However, I was not aware that there was any specific standard regarding how many errors should be signaled for a particular *invalid* UTF-8 string. (RFC 3629 just specifies how *valid* UTF-8 strings should be handled, not invalid ones.) In general, we want mbstring to comply as closely as possible with all relevant standards and to be compatible with as many legacy computing systems as possible. Hence, I am very happy to revert this change. This time around, I would also like to 1) add some code comments explaining why error handling for UTF-8 is done the way it is, and 2) add another test to our test suite specifically to verify the error handling behavior for UTF-8. To that end, I would like to ask if you have any other test cases which you can suggest should be added to the test suite. This will help to ensure that future maintenance and ongoing development of mbstring will not cause any further breakage.

« previous php.bugs (#240944) next »