Bug #70609 [NEW]: mbstring: simple casemaps instead Unicode's recommendation: full casemaps

From: Date: Wed, 30 Sep 2015 13:59:25 +0000
Subject: Bug #70609 [NEW]: mbstring: simple casemaps instead Unicode's recommendation: full casemaps
Groups: php.bugs 
Request: Send a blank email to php-bugs+get-196317@lists.php.net to get a copy of this message 
From:             cl at exomail dot to
Operating system: all
PHP version:      Irrelevant
Package:          mbstring related
Bug Type:         Bug
Bug description:mbstring: simple casemaps instead Unicode's recommendation: full casemaps

Description:
------------
mbstring's mb_strtoupper (and partners) do "simple casemapping" instead
of the recommended full casemapping.

http://www.unicode.org/versions/Unicode8.0.0/

Section 4.2 "Case", Headline "Case Mapping"
"The single-character mappings in UnicodeData.txt are insufficient for
languages such as German. Therefore, only legacy implementations that
cannot handle case mappings that increase string lengths should use
UnicodeData.txt case mappings alone."

PHP should stop being "legacy" and "insufficient".

For implementation guidelines, see section 5.18 "Case Mappings" and
ftp://ftp.unicode.org/Public/UCD/latest/ucd/SpecialCasing.txt


PHP's current implementation: 
php-src/ext/mbstring/ucgendat/ucgendat.c
seems to ignore "SpecialCasing.txt". 
The data structure
 static const unsigned int _uccase_map[]
in php-src/ext/mbstring/unicode_data.h assumes (IMHO) a one-to-one
mapping.

BTW:
Perl, Python, Java and others all do full case mapping.

This is *not* a bleeding edge feature. Perl has this since version 5.8.0
(year 2002!!!).


Test script:
---------------
$str="\xc3\x9f"; echo $str." upper =>
".mb_strtoupper($str,'UTF-8')."\n";
$str="\xc5\x89"; echo $str." upper =>
".mb_strtoupper($str,'UTF-8')."\n";


Expected result:
----------------
ß upper => SS
ʼn upper => ʼN


Actual result:
--------------
ß upper => ß
ʼn upper => ʼn


-- 
Edit bug report at https://bugs.php.net/bug.php?id=70609&edit=1
-- 
Try a snapshot (PHP 5.4):   https://bugs.php.net/fix.php?id=70609&r=trysnapshot54
Try a snapshot (PHP 5.5):   https://bugs.php.net/fix.php?id=70609&r=trysnapshot55
Try a snapshot (trunk):     https://bugs.php.net/fix.php?id=70609&r=trysnapshottrunk
Fixed in SVN:               https://bugs.php.net/fix.php?id=70609&r=fixed
Fixed in release:           https://bugs.php.net/fix.php?id=70609&r=alreadyfixed
Need backtrace:             https://bugs.php.net/fix.php?id=70609&r=needtrace
Need Reproduce Script:      https://bugs.php.net/fix.php?id=70609&r=needscript
Try newer version:          https://bugs.php.net/fix.php?id=70609&r=oldversion
Not developer issue:        https://bugs.php.net/fix.php?id=70609&r=support
Expected behavior:          https://bugs.php.net/fix.php?id=70609&r=notwrong
Not enough info:            https://bugs.php.net/fix.php?id=70609&r=notenoughinfo
Submitted twice:            https://bugs.php.net/fix.php?id=70609&r=submittedtwice
register_globals:           https://bugs.php.net/fix.php?id=70609&r=globals
PHP 4 support discontinued: https://bugs.php.net/fix.php?id=70609&r=php4
Daylight Savings:           https://bugs.php.net/fix.php?id=70609&r=dst
IIS Stability:              https://bugs.php.net/fix.php?id=70609&r=isapi
Install GNU Sed:            https://bugs.php.net/fix.php?id=70609&r=gnused
Floating point limitations: https://bugs.php.net/fix.php?id=70609&r=float
No Zend Extensions:         https://bugs.php.net/fix.php?id=70609&r=nozend
MySQL Configuration Error:  https://bugs.php.net/fix.php?id=70609&r=mysqlcfg

Thread (3 messages)

« previous php.bugs (#196317) next »