Bug #70609 [Opn->Csd]: mbstring: simple casemaps instead Unicode's recommendation: full casemaps

Edit report at https://bugs.php.net/bug.php?id=70609&edit=1 ID: 70609 Updated by: nikic@php.net Reported by: cl at exomail dot to Summary: mbstring: simple casemaps instead Unicode's recommendation: full casemaps -Status: Open +Status: Closed Type: Bug Package: mbstring related Operating System: all PHP Version: Irrelevant -Assigned To: +Assigned To: nikic Block user comment: N Private report: N Previous Comments: ------------------------------------------------------------------------ [2017-07-28 10:44:03] nikic@php.net I've implemented full case mapping in master, with the exception that the Final_Sigma condition is not handled. Relevant commit: https://github.com/php/php-src/commit/582a65b06f3de125887cab02d5c561168fcf94bc ------------------------------------------------------------------------ [2017-07-23 21:26:07] nikic@php.net Related To: Bug #70475 ------------------------------------------------------------------------ [2015-09-30 13:59:23] cl at exomail dot to Description: ------------ mbstring's mb_strtoupper (and partners) do "simple casemapping" instead of the recommended full casemapping. http://www.unicode.org/versions/Unicode8.0.0/ Section 4.2 "Case", Headline "Case Mapping" "The single-character mappings in UnicodeData.txt are insufficient for languages such as German. Therefore, only legacy implementations that cannot handle case mappings that increase string lengths should use UnicodeData.txt case mappings alone." PHP should stop being "legacy" and "insufficient". For implementation guidelines, see section 5.18 "Case Mappings" and ftp://ftp.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt PHP's current implementation: php-src/ext/mbstring/ucgendat/ucgendat.c seems to ignore "SpecialCasing.txt". The data structure static const unsigned int _uccase_map[] in php-src/ext/mbstring/unicode_data.h assumes (IMHO) a one-to-one mapping. BTW: Perl, Python, Java and others all do full case mapping. This is *not* a bleeding edge feature. Perl has this since version 5.8.0 (year 2002!!!). Test script: --------------- $str="\xc3\x9f"; echo $str." upper => ".mb_strtoupper($str,'UTF-8')."\n"; $str="\xc5\x89"; echo $str." upper => ".mb_strtoupper($str,'UTF-8')."\n"; Expected result: ---------------- ß upper => SS ʼn upper => ʼN Actual result: -------------- ß upper => ß ʼn upper => ʼn ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=70609&edit=1