Bug #80360 [Opn->Nab]: Unicode character property \p{L} truncates match

From: Date: Fri, 13 Nov 2020 13:39:57 +0000
Subject: Bug #80360 [Opn->Nab]: Unicode character property \p{L} truncates match
References: 1  Groups: php.bugs 
Request: Send a blank email to php-bugs+get-230326@lists.php.net to get a copy of this message 
Edit report at https://bugs.php.net/bug.php?id=80360&edit=1

 ID:                 80360
 Updated by:         nikic@php.net
 Reported by:        benjamin dot morel at gmail dot com
 Summary:            Unicode character property \p{L} truncates match
-Status:             Open
+Status:             Not a bug
 Type:               Bug
 Package:            PCRE related
 Operating System:   Ubuntu 18
 PHP Version:        7.4.12
 Block user comment: N
 Private report:     N

 New Comment:

You are missing the /u modifier. Your input is not being treated as UTF-8.


Previous Comments:
------------------------------------------------------------------------
[2020-11-13 13:37:32] benjamin dot morel at gmail dot com

Actually, "déjà" should be considered a single word with \p{L}, so the expected
result should be:

566f696cc3a02064c3a96ac3a0 true
566f696cc3a0 true
64c3a96ac3a0 true

------------------------------------------------------------------------
[2020-11-13 13:34:37] benjamin dot morel at gmail dot com

Description:
------------
When using \p{L} on accented letters, the resulting match is truncated and therefore invalid UTF-8.

Examples:

à = C3A0 is returned as C3
é = C3A9 is returned as C3


Test script:
---------------
$str = 'Voilà déjà';

display($str);

preg_match_all('/\p{L}+/', $str, $matches);

foreach ($matches[0] as $match) {
    display($match);
}

function display($str) {
    echo bin2hex($str), ' ';
    var_export(mb_check_encoding($str, 'UTF-8'));
    echo PHP_EOL;
}

Expected result:
----------------
566f696cc3a02064c3a96ac3a0 true
566f696cc3a0 true
64c3a9 true
6ac3a0 true


Actual result:
--------------
566f696cc3a02064c3a96ac3a0 true
566f696cc3 false
64c3 false
6ac3 false



------------------------------------------------------------------------



--
Edit this bug report at https://bugs.php.net/bug.php?id=80360&edit=1

Thread (4 messages)

« previous php.bugs (#230326) next »