Edit report at https://bugs.php.net/bug.php?id=62010&edit=1
ID: 62010
Updated by: bukka@php.net
Reported by: tklingenberg at lastflood dot net
Summary: json_decode produces invalid byte-sequences
Status: Assigned
Type: Bug
Package: JSON related
Operating System: Windows
PHP Version: 5.3.13
Assigned To: bukka
Block user comment: N
Private report: N
New Comment:
Just a small correction. The ABNF does not prohibit single unpaired UTF-16
surrogate (not surrogate sequence of course).
Previous Comments:
------------------------------------------------------------------------
[2015-05-31 20:04:01] bukka@php.net
I think you don't understand the RFC and the string ABNF. Please read it carefully once
again... mainly https://tools.ietf.org/html/rfc7159#section-7
You will see this ABNF:
string = quotation-mark *char quotation-mark
char = unescaped /
escape (
%x22 / ; " quotation mark U+0022
%x5C / ; \ reverse solidus U+005C
%x2F / ; / solidus U+002F
%x62 / ; b backspace U+0008
%x66 / ; f form feed U+000C
%x6E / ; n line feed U+000A
%x72 / ; r carriage return U+000D
%x74 / ; t tab U+0009
%x75 4HEXDIG ) ; uXXXX U+XXXX
escape = %x5C ; \
quotation-mark = %x22 ; "
unescaped = %x20-21 / %x23-5B / %x5D-10FFFF
This line specifically shows that: %x75 4HEXDIG
It doesn't say anything about prohibiting surrogate sequence for unicode escape. Actually the
note about that is the section 8.2 that I quoted before. Especially this sentence says that:
However, the ABNF in this specification allows member names and
string values to contain bit sequences that cannot encode Unicode
characters; for example, "\uDEAD" (a single unpaired UTF-16
surrogate).
That is all about escaped sequences (see "\uDEAD"...). Of cource, binary strings have to
be correctly encoded in UTF-8 (the only supported input encoding for PHP json parser) as stated in
section 8.1.
I understand that such unicode escapes might be inconvinient and that's why I emailed internals
about introducing new constant for it that will address your issue. I plan to merge it to master
next week but it will be just non-default option as we have to have RFC complained parser.
I actually had it already implemented as a default when I was rewritting json for PHP 7. Then I
noticed that the RFC says this and I removed it ( https://github.com/php/php-src/commit/1119c4d2b210730e6c26f034e9769d116b26ceb1
). I changed it in jsond as well but in this case I added the constant which is what I plan to add
to json now.
------------------------------------------------------------------------
[2015-05-31 15:42:49] tklingenberg at lastflood dot net
Hi bukka,
thank you for taking the time to look into it.
But I'm very sorry to highlight that the information you've provided in your comment is
best of all only remotely related to this issue and does not touch the root-cause of the flaw
reported here.
You've perhaps been misguided by the internals mailings (haven't read those), the part you
quote is about binary string data.
But the report I created is *not* about binary data, you can see, the string presented is US-ASCII
without any control characters.
It's about the strings _represented_ by JSON (not in binary), and more specifically the option
to use an \uXXXX (six characters) escape sequence for any character in the Basic Multilingual Plane
(U+0000 through U+FFFF). Please see Section 7 of the JSON RFC.
U+D834 is not a character in the Basic Multilingual Plane (see Unicode, compare with a reference,
exemplary: http://www.fileformat.info/info/unicode/char/d834/index.htm).
If a string would have been passed json_decode containing the related binary sequence - as what you
say would be allowed by the JSON spec - PHP handles it correctly according the documented contract:
The binary sequence would qualify as *not* being an UTF-8 string and therefore the result of the
function is unexpected:
<?php
$notUTF8 = "\"\xED\xA0\xB4\"";
var_dump($notUTF8); // string(5) ""���""
$result = json_decode($notUTF8);
var_dump($result); // NULL
That's covered by the specs you quote, but not the flaw I reported here. As you can see, this
is a different example and I can't see that PHP violates the spec nor it's own contract
here.
------------------------------------------------------------------------
[2015-05-28 18:58:45] bukka@php.net
I just emailed about this on internals. This is not a bug as it is conformant with the JSON RFC 7159
as noted in section 8.2:
However, the ABNF in this specification allows member names and
string values to contain bit sequences that cannot encode Unicode
characters; for example, "\uDEAD" (a single unpaired UTF-16
surrogate). Instances of this have been observed, for example, when
a library truncates a UTF-16 string without checking whether the
truncation split a surrogate pair. The behavior of software that
receives JSON texts containing such values is unpredictable; for
example, implementations might return different values for the length
of a string value or even suffer fatal runtime exceptions.
As you can see that behavior is unpredictable.
However I see a use case here and that's why I proposed new option JSON_VALID_ESCAPED_UNICODE
that would emit JSON_ERROR_UTF16 if such sequence appears in decoded string.
------------------------------------------------------------------------
[2013-07-12 15:55:24] masakielastic at gmail dot com
Here is RFC 3629's description about UTF-8 definition.
The definition of UTF-8 prohibits encoding character numbers
between U+D800 and U+DFFF, which are reserved for use with the
UTF-16 encoding form (as surrogate pairs) and do not directly
represent characters.
http://tools.ietf.org/html/rfc3629
The following patch solve the part of problem,
The isolated low surrogate pairs(U+DC00 U+DFFF) are replaced with U+FFFD,
The imrovement for high surrogate pairs (U+D800 - U+DBFF) is needed.
https://gist.github.com/masakielastic/5985383
var_dump(
"\xef\xbf\xbd" === json_decode('"\udc00"'),
"\xef\xbf\xbd"."\xed\xa0\x80" ===
json_decode('"\ud800\ud800"'),
"\xed\xa0\x80" === json_decode('"\ud800"')
);
The consistency for the following options
(under the discussion) is needed too.
json_encode's option for replacing ill-formd byte sequences
with substitute characters
https://bugs.php.net/bug.php?id=65082
------------------------------------------------------------------------
[2013-01-11 09:44:55] votefordevnull at gmail dot com
Successfully reproduced on Linux
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=62010
--
Edit this bug report at https://bugs.php.net/bug.php?id=62010&edit=1