[php-src] Issue #8360: mbstring: UTF-8 sequences with out-of-bounds follow bytes are no longer substituted correctly

From: Date: Wed, 13 Apr 2022 10:26:21 +0000
Subject: [php-src] Issue #8360: mbstring: UTF-8 sequences with out-of-bounds follow bytes are no longer substituted correctly
Groups: php.bugs 
Request: Send a blank email to php-bugs+get-240917@lists.php.net to get a copy of this message
Issue: https://github.com/php/php-src/issues/8360 Author: ausi ### Description The following code: ```php <?php mb_substitute_character(ord('?')); var_dump(mb_convert_encoding("\xDF\xC0", 'UTF-8', 'UTF-8')); var_dump(mb_convert_encoding("\xEF\xBF\xC0", 'UTF-8', 'UTF-8')); var_dump(mb_convert_encoding("\xF4\x8F\xBF\xC0", 'UTF-8', 'UTF-8')); ``` Resulted in this output: ``` string(1) "?" string(1) "?" string(1) "?" ``` But I expected this output instead (as it is in PHP 7.x and 8.0): ``` string(2) "??" string(2) "??" string(2) "??" ``` See https://3v4l.org/Qt2HJ As far as I know mb_convert_encoding(…, 'UTF-8', 'UTF-8') is commonly used to convert (potentially) non-conforming UTF-8 to valid UTF-8. Before PHP 8.1.0 this worked as expected and behaved the same way as all major web browsers do. But since PHP 8.1.0 the behavior has changed and UTF-8 sequences with out-of-bounds follow bytes are no longer substituted with two substitute characters (usually represented as �) but with only one. To my understanding this is defined in the encoding specification: <https://encoding.spec.whatwg.org/#utf-8-decoder> The step 8.1.1.4.2 _“Prepend byte to ioQueue.”_ seems to be missing in these cases. ### PHP Version PHP 8.1.5 (all 8.1.x are affected) ### Operating System _No response_

« previous php.bugs (#240917) next »