#26623 [Opn->Bgs]: XML Parser ignores UTF-8 input encoding?

ID: 26623 User updated by: steven at acko dot net Reported By: steven at acko dot net -Status: Open +Status: Bogus Bug Type: XML related Operating System: Windows 2000 PHP Version: 4CVS-2003-12-14 (stable) New Comment: Hmmm I finally figured this out: PHP4 apparently always expects you to specify the input encoding manually as a parameter to xml_parser_create(). The manual is a bit unclear about this (it says "you /can/ specify an input encoding" rather than "you have to"). In PHP5, the parameter to xml_parser_create() is ignored completely, and the XML parser extract the encoding on its own. Previous Comments: ------------------------------------------------------------------------ [2003-12-15 16:43:10] steven at acko dot net This bug does not happen with the latest PHP5 by the way. PHP5 will correctly handle my example. ------------------------------------------------------------------------ [2003-12-14 23:13:35] steven at acko dot net Description: ------------ PHP seems to ignore the encoding when parsing an UTF-8 encoded XML file, and assumes it is ISO-8859-1 instead. The code below contains a very short XML file (inline) to parse, with the character a-with-tilde as value (this is just to show what happens, there is nothing special about this character). The a-with-tilde takes 2 bytes in UTF-8 encoding, and is represented as such in the XML file string. The buggy behaviour is illustrated with the 3 possible PHP output encodings. Comment/uncomment the correct xml_parser_set_option() call to see the behaviour in all output encodings. Note that nothing changes in behaviour if you change the encoding="utf-8" in the source XML into 'iso-8859-1', or remove it altogether, which shows that it is being ignored. Reproduce code: --------------- <?php $xmlfile = "<?xml version=\"1.0\" encoding=\"utf-8\" ?><tag>\xC3\xA3</tag>"; function handler_data($parser, $data) { print "Data: $data\n"; print "Length: ". strlen($data) ."\n"; } $xml_parser = xml_parser_create(); xml_set_character_data_handler($xml_parser, "handler_data"); xml_parser_set_option($xml_parser, XML_OPTION_TARGET_ENCODING, "utf-8"); // xml_parser_set_option($xml_parser, XML_OPTION_TARGET_ENCODING, "iso-8859-1"); // xml_parser_set_option($xml_parser, XML_OPTION_TARGET_ENCODING, "us-ascii"); xml_parse($xml_parser, $xmlfile, 1); xml_parser_free($xml_parser); ?> Expected result: ---------------- With output encoding set to UTF-8, PHP should leave the data alone and should print 2 bytes (\xC3 and \xA3) for the a-with-tilde. With output encoding set to ISO-8859-1, PHP should encode the a-with-tile as a single byte (\xE3). With output encoding set to US-ASCII, PHP should output a single '?' to indicate the a-with-tilde is not available in the output encoding. Actual result: -------------- With output encoding set to UTF-8, PHP re-encodes the input into UTF-8, resulting in 4 bytes for a-with-tilde. With output encoding set to ISO-8859-1, PHP leaves the input untouched. With output encoding set to US-ASCII, PHP outputs two question marks. ------------------------------------------------------------------------ -- Edit this bug report at http://bugs.php.net/?id=26623&edit=1