Bug #70609 [Opn]: mbstring: simple casemaps instead Unicode's recommendation: full casemaps

From: Date: Fri, 28 Jul 2017 10:44:09 +0000
Subject: Bug #70609 [Opn]: mbstring: simple casemaps instead Unicode's recommendation: full casemaps
References: 1  Groups: php.bugs 
Request: Send a blank email to php-bugs+get-210394@lists.php.net to get a copy of this message 
Edit report at https://bugs.php.net/bug.php?id=70609&edit=1

 ID:                 70609
 Updated by:         nikic@php.net
 Reported by:        cl at exomail dot to
 Summary:            mbstring: simple casemaps instead Unicode's
                     recommendation: full casemaps
 Status:             Open
 Type:               Bug
 Package:            mbstring related
 Operating System:   all
 PHP Version:        Irrelevant
 Block user comment: N
 Private report:     N

 New Comment:

I've implemented full case mapping in master, with the exception that the Final_Sigma condition
is not handled.

Relevant commit: https://github.com/php/php-src/commit/582a65b06f3de125887cab02d5c561168fcf94bc


Previous Comments:
------------------------------------------------------------------------
[2017-07-23 21:26:07] nikic@php.net

Related To: Bug #70475

------------------------------------------------------------------------
[2015-09-30 13:59:23] cl at exomail dot to

Description:
------------
mbstring's mb_strtoupper (and partners) do "simple casemapping" instead of the
recommended full casemapping.

http://www.unicode.org/versions/Unicode8.0.0/

Section 4.2 "Case", Headline "Case Mapping"
"The single-character mappings in UnicodeData.txt are insufficient for languages such as
German. Therefore, only legacy implementations that cannot handle case mappings that increase string
lengths should use UnicodeData.txt case mappings alone."

PHP should stop being "legacy" and "insufficient".

For implementation guidelines, see section 5.18 "Case Mappings" and
ftp://ftp.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt


PHP's current implementation: 
php-src/ext/mbstring/ucgendat/ucgendat.c
seems to ignore "SpecialCasing.txt". 
The data structure
 static const unsigned int _uccase_map[]
in php-src/ext/mbstring/unicode_data.h assumes (IMHO) a one-to-one mapping.

BTW:
Perl, Python, Java and others all do full case mapping.

This is *not* a bleeding edge feature. Perl has this since version 5.8.0 (year 2002!!!).


Test script:
---------------
$str="\xc3\x9f"; echo $str." upper =>
".mb_strtoupper($str,'UTF-8')."\n";
$str="\xc5\x89"; echo $str." upper =>
".mb_strtoupper($str,'UTF-8')."\n";


Expected result:
----------------
ß upper => SS
ʼn upper => ʼN


Actual result:
--------------
ß upper => ß
ʼn upper => ʼn



------------------------------------------------------------------------



--
Edit this bug report at https://bugs.php.net/bug.php?id=70609&edit=1

Thread (3 messages)

« previous php.bugs (#210394) next »