#41067 [Opn->Fbk]: JSON_decode() does not handle surrogate pairs

ID: 41067 Updated by: iliaa@php.net Reported By: jp at df5ea dot net -Status: Open +Status: Feedback Bug Type: *Unicode Issues PHP Version: 5CVS-2007-04-12 (CVS) New Comment: Can you please provide an optimized version of the patch? Previous Comments: ------------------------------------------------------------------------ [2007-04-12 20:07:52] jp at df5ea dot net http://anna.df5ea.net/~jp/JSON_parser.c.patch An extra parameter is added to utf16_to_utf8(): prev_utf16. This parameter is used to store the previously decoded UTF-16 code unit. When the function encounters an high surrogate this value is used to look for a low surrogate. From this pair it builds the correct UTF-8 sequence. When it encounters an surrogate code point not in a pair it is ignored. The prev_utf16 variable in JSON_parser() is reset between different strings. If there is a speed concern regarding the parser it is also possible to drop the prev_utf16 part. The decoder function could then look in the decoding buffer to look for the low surrogate. If needed I can submit a patch to get the function operating in this way. ------------------------------------------------------------------------ [2007-04-12 19:41:17] iliaa@php.net Can you post a link to the patch? ------------------------------------------------------------------------ [2007-04-12 18:12:28] jp at df5ea dot net Description: ------------ When decoding a string with surrogate pairs in it, JSON_decode() produces incorrect UTF-8. Instead of encoding the two surrogate characters as one UTF-8 sequence it encodes it as two sequences wich represent the two surrogate code points. The decoded string is actually CESU-8. The JSON_encode() function can not encode such a string. I have a patch to JSON_parse.c that transcodes the UTF-16 properly to UTF-8. Reproduce code: --------------- <?php $single_barline = "\360\235\204\200"; $array = array($single_barline); print bin2hex($single_barline) . "\n"; // print $single_barline . "\n\n"; $json = json_encode($array); print $json . "\n\n"; $json_decoded = json_decode($json, true); // print $json_decoded[0] . "\n"; print bin2hex($json_decoded[0]) . "\n"; print "END\n"; ?> Expected result: ---------------- The output form the two bin2hex functions should be the same: f09d8480 ["\ud834\udd00"] f09d8480 END Actual result: -------------- The second string is different from the input string and illegal UTF-8. f09d8480 ["\ud834\udd00"] eda0b4edb480 END ------------------------------------------------------------------------ -- Edit this bug report at http://bugs.php.net/?id=41067&edit=1