Edit report at https://bugs.php.net/bug.php?id=65082&edit=1
ID: 65082
Updated by: bukka@php.net
Reported by: masakielastic at gmail dot com
Summary: json option for replacing ill-formd byte sequences
with substitute char
-Status: Assigned
+Status: Closed
Type: Feature/Change Request
Package: JSON related
Operating System: All
PHP Version: 5.5.0
Assigned To: bukka
Block user comment: N
Private report: N
New Comment:
This has been addressed and will be part of PHP 7.2
Previous Comments:
------------------------------------------------------------------------
[2017-06-17 10:15:24] bukka@php.net
I'm working on it and the first part (json_encode) can be seen in here:
https://github.com/bukka/php-src/commit/0a5fe6e0dc709985752ac82794f0f45de0ead26f
------------------------------------------------------------------------
[2013-07-22 05:09:35] masakielastic at gmail dot com
I created a repo for the patches and the report of benchmarks
https://github.com/masakielastic/patches/tree/master/php_bugs_65082
The difference between json_utf8_to_utf16 and json_utf8_to_utf32 isn't seen.
the use of json_utf8_to_utf32 or the direct use of php_next_utf8_char
in json_escape_string is better choice for
JSON_NOTUTF8_SUBSTITUTE and JSON_NOTUTF8_SUBSTITUTE|JSON_UNESCAPED_UNICODE.
php_next_utf8_char in json_escape_string is a bit faster than
json_utf8_to_utf32 for JSON_NOTUTF8_SUBSTITUTE.
https://github.com/masakielastic/patches/blob/master/php_bugs_65082/04_php_next_
utf8_char_in_json_escape_string.patch
https://github.com/masakielastic/patches/blob/master/php_bugs_65082/04_php_next_
utf8_char_in_json_escape_string.c
------------------------------------------------------------------------
[2013-07-19 16:46:49] masakielastic at gmail dot com
Another way of perfomance improvemnet is using php_next_utf8_char directly
in json_escape_string on the condition of PHP_JSON_NOTUTF8_SUBSTITUTE
and PHP_JSON_NOTUTF8_IGNORE.
This way reduces one loop compared with using json_utf8_to_utf16.
------------------------------------------------------------------------
[2013-07-19 16:33:24] masakielastic at gmail dot com
I agree with you on isolated surrogate pairs.
The test cases for json_decode and JSON_NOTUTF8_SUBSTITUTE and
JSON_NOTUTF8_IGNORE must be contained
since json_decode uses json_utf8_to_utf16.
https://github.com/php/php-src/blob/master/ext/json/json.c#L673
I already posted the test cases.
https://gist.github.com/masakielastic/5973095#file-04-test-php-L26
"a\xEF\xBF\xBD" === json_decode('"'."a\x80".'"',
false, 512,
JSON_NOTUTF8_SUBSTITUTE),
"a" === json_decode('"'."a\x80".'"', false, 512,
JSON_NOTUTF8_IGNORE)
The one way of perfomance improvement is adding json_utf8_to_utf32.
I posted another patch.
https://gist.github.com/masakielastic/5973095#file-02-json_unescaped_unicode-
patch
I created unsigned int *utf32 data type
for not changing unsigned short *utf16 data type.
If you want to provide a common variable
for json_utf8_to_utf16 and json_utf8_to_utf32,
the modification for JSON_parser.c is also needed.
The one of candidate for the name of variable is
unsigned int *code_codes.
http://www.unicode.org/glossary/#code_unit
I also updated the previous patch.
https://gist.github.com/masakielastic/5973095#file-01-json_unescaped_unicode-
patch
if (options & PHP_JSON_UNESCAPED_UNICODE) {
+ if (us < 0x20) {
+ smart_str_appendl(buf, "\\u", 2);
+ smart_str_appendc(buf, digits[(us >> 12) & 0xf]);
+ smart_str_appendc(buf, digits[(us >> 8) & 0xf]);
+ smart_str_appendc(buf, digits[(us >> 4) & 0xf]);
+ smart_str_appendc(buf, digits[(us & 0xf)]);
+ } else if (us < 0x80) {
------------------------------------------------------------------------
[2013-07-15 07:31:49] remi@php.net
> Hi remi, could you test my patch for PHP_JSON_UNESCAPED_UNICODE option?
> The patch adopts JSON_NOTUTF8_SUBSTITUTE and JSON_NOTUTF8_IGNORE options.
The PHP_JSON_UNESCAPED_UNICODE + JSON_NOTUTF8_IGNORE already works with my patch.
Yes, PHP_JSON_UNESCAPED_UNICODE + JSON_NOTUTF8_SUBSTITUTE doesn't work for now, but converting
to utf16, then back to utf8 seems really... messy. Need something simpler.
Notice: this bug is only for json_encode. Other issue have their own bug for tracking (especially
the json_decode one, as I dont plan to alter it)
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=65082
--
Edit this bug report at https://bugs.php.net/bug.php?id=65082&edit=1