Doc #78245 [Com]: preg_split('/\R/', 'Техни') wrongly splits text into 2 parts

From: Date: Tue, 03 Oct 2023 05:27:06 +0000
Subject: Doc #78245 [Com]: preg_split('/\R/', 'Техни') wrongly splits text into 2 parts
References: 1  Groups: php.doc.bugs 
Request: Send a blank email to doc-bugs+get-19669@lists.php.net to get a copy of this message 
Edit report at https://bugs.php.net/bug.php?id=78245&edit=1

 ID:                 78245
 Comment by:         tannaowens6 at googlemail dot com
 Reported by:        bugs_php at zarevak dot net
 Summary:            preg_split('/\R/', 'Техни') wrongly splits
text
                     into 2 parts
 Status:             Verified
 Type:               Documentation Problem
 Package:            PCRE related
 Operating System:   any (tested Windows, Linux)
 PHP Version:        7.3.6
 Block user comment: N
 Private report:     N

 New Comment:

Thank you for this information it is very helpful for me
nice.(https://github.com)(https://www.wellstarsmartsquare.com/)


Previous Comments:
------------------------------------------------------------------------
[2019-07-03 14:02:37] danack@php.net

For anyone looking to update the manual, this section from the PCRE docs is probably relevant, and
non-trivial: https://www.pcre.org/original/doc/html/pcrepattern.html#newlineseq

"Newline sequences 

Outside a character class, by default, the escape sequence \R matches any Unicode newline sequence.
In 8-bit non-UTF-8 mode \R is equivalent to the following:

  (?>\r\n|\n|\x0b|\f|\r|\x85)

This is an example of an "atomic group", details of which are given below. This particular
group matches either the two-character sequence CR followed by LF, or one of the single characters
LF (linefeed, U+000A), VT (vertical tab, U+000B), FF (form feed, U+000C), CR (carriage return,
U+000D), or NEL (next line, U+0085). The two-character sequence is treated as a single unit that
cannot be split.
In other modes, two additional characters whose codepoints are greater than 255 are added: LS (line
separator, U+2028) and PS (paragraph separator, U+2029). Unicode character property support is not
needed for these characters to be recognized.

It is possible to restrict \R to match only CR, LF, or CRLF (instead of the complete set of Unicode
line endings) by setting the option PCRE_BSR_ANYCRLF either at compile time or when the pattern is
matched. (BSR is an abbrevation for "backslash R".) This can be made the default when PCRE
is built; if this is the case, the other behaviour can be requested via the PCRE_BSR_UNICODE option.
It is also possible to specify these settings by starting a pattern string with one of the following
sequences:

  (*BSR_ANYCRLF)   CR, LF, or CRLF only
  (*BSR_UNICODE)   any Unicode newline sequence

These override the default and the options given to the compiling function, but they can themselves
be overridden by options given to a matching function. Note that these special settings, which are
not Perl-compatible, are recognized only at the very start of a pattern, and that they must be in
upper case. If more than one of them is present, the last one is used. They can be combined with a
change of newline convention; for example, a pattern can start with:

  (*ANY)(*BSR_ANYCRLF)

They can also be combined with the (*UTF8), (*UTF16), (*UTF32), (*UTF) or (*UCP) special sequences.
Inside a character class, \R is treated as an unrecognized escape sequence, and so matches the
letter "R" by default, but causes an error if PCRE_EXTRA is set."

------------------------------------------------------------------------
[2019-07-03 13:59:45] cmb@php.net

From the PCRE2 docs[1]:

| In 8-bit non-UTF-8 mode \R is equivalent to the following:
|
|  (?>\r\n|\n|\x0b|\f|\r|\x85)

[1] <https://www.pcre.org/current/doc/html/pcre2pattern.html#newlineseq>

------------------------------------------------------------------------
[2019-07-03 13:54:18] nikic@php.net

\R matches any Unicode line break. To only match \r, \n and \r\n the (*BSR_ANYCRLF) mode needs to be
used.

------------------------------------------------------------------------
[2019-07-03 13:50:06] bugs_php at zarevak dot net

Description:
------------
According to https://www.php.net/manual/en/regexp.reference.escape.php
the \R should match line-breaks (\n, \r and \r\n), but here it matches something else and breaks the
string into two. When using expression \r\n|\n|\r directly, it works correctly.

Adding 'u' pattern modifier fixes the issue, but checks validity of the incoming UTF-8
string, which I do not want. The Russian text 'Техни' in this example does
not contain \r or \n characters even when encoded using UTF-8 so this should not apply. The \R
splits the string in the middle of the 'х' (U+0445 CYRILLIC SMALL LETTER HA) character,
which is encoded in UTF-8 as bytes #D1 #85.

This is either bug:
1] in implementation of \R, where it incorrectly matches different bytes then 13, 10 and their
combination
2] or in documentation, where it should state what other characters it can match (in this example it
seems, it matches U+0085 <control> : NEXT LINE [NEL])

Test script:
---------------
<?php
$string="Реконструкция\r\nРеконструкция
- Служба Технической
поддержки\r\n";
$array = preg_split('/\R/', $string);  // BUG!
$array2 = preg_split('/\r\n|\n|\r/', $string); //OK :)
var_dump($array, $array2, ($array===$array2));

Expected result:
----------------
array(3) {
  [0]=>
  string(26) "Реконструкция"
  [1]=>
  string(83) "Реконструкция - Служба
Технической поддержки"
  [2]=>
  string(0) ""
}
array(3) {
  [0]=>
  string(26) "Реконструкция"
  [1]=>
  string(83) "Реконструкция - Служба
Технической поддержки"
  [2]=>
  string(0) ""
}
bool(true)

Actual result:
--------------
array(4) {
  [0]=>
  string(26) "Реконструкция"
  [1]=>
  string(47) "Реконструкция - Служба
Те"
  [2]=>
  string(35) "нической поддержки"
  [3]=>
  string(0) ""
}
array(3) {
  [0]=>
  string(26) "Реконструкция"
  [1]=>
  string(83) "Реконструкция - Служба
Технической поддержки"
  [2]=>
  string(0) ""
}
bool(false)


------------------------------------------------------------------------



--
Edit this bug report at https://bugs.php.net/bug.php?id=78245&edit=1

Thread (4 messages)

« previous php.doc.bugs (#19669) next »