[php-src] Issue #8279: mb_detect_encoding does not return the first matching encoding anymore

Issue: https://github.com/php/php-src/issues/8279 Comment Author: alexdowad > It seems the behavior of mb_detect_encoding changed in PHP 8.1, not clear if this is on purpose > or not. It's on purpose. As you say, previously mb_detect_encoding would just return the first listed encoding which 'works' for the input string. Now it also applies heuristics to detect which of the valid text encodings in the specified list (if there are more than one) is most likely to be correct. If you really, really just want to know "which is the first text encoding in *this* list which is valid for *this* string", it would be better to call mb_check_encoding in a loop. If the docs need to be updated, I would like to suggest you could open a GitHub issue for any inaccurate statements you found there. Now let's see the reason for the difference you found. First of all, your string is valid in both UTF-8 and ISO-8859-1. In UTF-8, the bytes break down as: ``2f 64 61 76 2f 66 69 6c 65 73 2f 61 64 6d 69 6e 2f c3bc c3b6 c3a4 c3b6 c3a4 c3bc c3b6 c3a4 c3bb c5b7 c3ae`` Whereas in ISO-8859-1, they break down in this (trivial) way: ``2f 64 61 76 2f 66 69 6c 65 73 2f 61 64 6d 69 6e 2f c3 bc c3 b6 c3 a4 c3 b6 c3 a4 c3 bc c3 b6 c3 a4 c3 bb c5 b7 c3 ae`` All those 2-byte UTF-8 characters can also be interpreted as two ISO-8859-1 characters, as you can see. The current implementation of mb_detect_encoding has a slight preference for text encodings in which the input string comes out to a shorter sequence of codepoints. However, what makes it prefer ISO-8859-1 over UTF-8 in this case is the letter ŷ. I don't know if that accented letter is commonly used in any major language of the world, but currently mb_detect_encoding classifies it as a "rare" character and in this case, penalizes UTF-8 because of it. If you remove ŷ from that string, mb_detect_encoding will decide that UTF-8 is more likely what you wanted. If the input string was longer, then UTF-8 would have much better chances of winning out. In general, trying to automatically detect text encoding on short strings is very error-prone. This is also true for the simpler approach of "just picking an encoding that works"; as the input string becomes shorter, the chances that an unintended encoding will match by accident become higher. If there are good reasons why mb_detect_encoding should consider ŷ to be a "common" character, please feel free to propose that.