#35647 [Opn->Fbk]: tidy does not produce valid utf8 when the encoding is specified in the config

From: Date: Mon, 12 Dec 2005 21:05:51 +0000
Subject: #35647 [Opn->Fbk]: tidy does not produce valid utf8 when the encoding is specified in the config
References: 1  Groups: php.bugs 
Request: Send a blank email to php-bugs+get-90309@lists.php.net to get a copy of this message 
 ID:               35647
 Updated by:       tony2001@php.net
 Reported By:      bugs at nikmakepeace dot com
-Status:           Open
+Status:           Feedback
 Bug Type:         XML related
 Operating System: FC3
 PHP Version:      5.1.1
 New Comment:

Put the data somewhere in the Net and paste the link here, please.



Previous Comments:
------------------------------------------------------------------------

[2005-12-12 18:44:35] bugs at nikmakepeace dot com

Description:
------------
If you specify utf8 encoding using the config options 'char-encoding',
'input-encoding' and 'output-encoding' with tidy it converts HTML
entities into their latin1, single-byte equivalents rather than the
correct, multi-byte utf-8 encodings (or just leaving them as entities)


The result is that   is converted into 0xA0, é is converted
into 0xE9 and so on. This is not valid UTF-8 and so well-behaving XML
parsers, including PHP's DOM, fail.

Specifying 'utf8' as the third parameter works correctly.

Reproduce code:
---------------
<?php
$dirty='<a
href="http://fr.yahoo.com/r/n/fd/7/*http://fr.news.yahoo.com/12122005/202/beatrice-dalle-temoigne-au-proces-de-son-mari-accuse-de.html">B&eacute;atrice
Dalle t&eacute;moigne au proc&egrave;s de son mari accus&eacute; de
viol</a><br/>
<small><nobr><a
href="http://rd.yahoo.co.jp/toppage/topinfo/rikunabi/051213/?http://katsuyou.rikunabi-shinsotsu.yahoo.co.jp/2007/">人と差がつく就職活動をしよう</a></nobr>
- <nobr><a
href="http://rd.yahoo.co.jp/toppage/topinfo/event_xmas/051125/?http://xmas.yahoo.co.jp/">ポイント5倍のクリスマスギフトは12時まで！</a></nobr></small>';

$config['char-encoding']='utf8';
$config['input-encoding']='utf8';
$config['output-encoding']='utf8';
$config['output-xhtml']=true;

echo tidy_repair_string($dirty, $config);
?>


Expected result:
----------------
Note well the correct unicode e-acute and e-grave in the French text.

<?xml version="1.0"?>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
</head>
<body>
<a href=
"http://fr.yahoo.com/r/n/fd/7/*http://fr.news.yahoo.com/12122005/202/beatrice-dalle-temoigne-au-proces-de-son-mari-accuse-de.html">
Béatrice Dalle témoigne au procès de son mari accusé de
viol</a><br />
<small><nobr><a href=
"http://rd.yahoo.co.jp/toppage/topinfo/rikunabi/051213/?http://katsuyou.rikunabi-shinsotsu.yahoo.co.jp/2007/">
人と差がつく就職活動をしよう</a></nobr>
- <nobr><a href=
"http://rd.yahoo.co.jp/toppage/topinfo/event_xmas/051125/?http://xmas.yahoo.co.jp/">
ポイント5倍のクリスマスギフトは12時まで！</a></nobr></small>
</body>
</html>


Actual result:
--------------
Note how the e-acute and e-grave has been replaced with a non-unicode
character.

<?xml version="1.0"?>
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title></title>
</head>
<body>
<a href=
"http://fr.yahoo.com/r/n/fd/7/*http://fr.news.yahoo.com/12122005/202/beatrice-dalle-temoigne-au-proces-de-son-mari-accuse-de.html">
B�atrice Dalle t�moigne au proc�s de son mari accus� de
viol</a><br />
<small><nobr><a href=
"http://rd.yahoo.co.jp/toppage/topinfo/rikunabi/051213/?http://katsuyou.rikunabi-shinsotsu.yahoo.co.jp/2007/">
人と差がつく就職活動をしよう</a></nobr>
-
<nobr><a href=
"http://rd.yahoo.co.jp/toppage/topinfo/event_xmas/051125/?http://xmas.yahoo.co.jp/">
ポイント5倍のクリスマスギフトは12時まで！</a></nobr></small>
</body>
</html>



------------------------------------------------------------------------


-- 
Edit this bug report at http://bugs.php.net/?id=35647&edit=1

Thread (5 messages)

« previous php.bugs (#90309) next »