Bug #81390 [Asn]: mb_detect_encoding() regression

Edit report at https://bugs.php.net/bug.php?id=81390&edit=1 ID: 81390 Updated by: alexdowad@php.net Reported by: alec at alec dot pl Summary: mb_detect_encoding() regression Status: Assigned Type: Bug Package: mbstring related PHP Version: 8.1.0beta5 Assigned To: alexdowad Block user comment: N Private report: N New Comment: Alec, you are nothing short of a fabulous tester. Well done. I am going to investigate your latest finding... just not right now. At the moment, I am cooking something up which will hopefully make almost every mbstring function several times faster. Soon to be served up, hot and fresh, in a upcoming PHP release. Bon appetit! Previous Comments: ------------------------------------------------------------------------ [2021-11-04 17:35:39] alec at alec dot pl After testing rc5 I have one last case. https://3v4l.org/9o7ft There is a reasonably short input ('Iksi' . chr(241) . 'ski') recognized as SJIS-2004, while I'd expect ISO-8859-2 or ISO-8859-1. It is also interesting that without the second argument the result would be UTF-8. I understand that such a short input may be confusing for the detector, but maybe there's still some bug. ------------------------------------------------------------------------ [2021-11-01 19:40:34] patrickallaert@php.net This is part of PHP 8.1.0RC5. ------------------------------------------------------------------------ [2021-11-01 19:35:11] alexdowad@php.net Closing this for now, as I believe the enhancements which have been merged in to mainline will fully address Alec's concerns. (Whenever a new public release comes out which he can install and try, that is.) Alec, please re-open this if you still discover any problems with the detection accuracy of mb_detect_encoding() in the next rc release. On the other hand, if it works well for you, please drop a line as well. ------------------------------------------------------------------------ [2021-10-17 18:48:44] alexdowad@php.net https://github.com/php/php-src/pull/7587 ------------------------------------------------------------------------ [2021-10-17 17:20:44] alexdowad@php.net Hi Alec, The test case which you provided in the original bug report does appear to return good results now. Looks like the bug fix had a good effect: https://3v4l.org/UXtBd Thank you for providing more test cases in which the latest implementation of mb_detect_encoding does not return accurate results. Those will be added to the test suite for the PHP interpreter. I have been working on another patch for mbstring which significantly enhances the accuracy of legacy encoding detection. Several of the test cases you have added in this thread already return the correct result with that new patch. Once the next patch is merged, we would appreciate your further assistance with testing. If you are able to build PHP from source, it would be even more helpful if you can test even before the next rc. However, I understand that building a C program may seem daunting for some who have never done that before, and it is work which you didn't sign up for. ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=81390 -- Edit this bug report at https://bugs.php.net/bug.php?id=81390&edit=1