note 103347 added to function.iconv

Data migration on Mysql with curly quotes in text fields can be VERY tricky sometimes. I'm not sure how my database got to be populated with these mysterious characters but here's what happened, in case anyone else benefits: The curly quotes both single and double, and the ellipses, all came out as boxed question-marks when printed on a UTF-8 encoded web page. If I forced the encoding to ISO-Latin-1 they looked fine. So you'd think they were ISO-Latin-1 quotes, eh? Not a chance. The following functions, getting increasingly desperate, had NO EFFECT AT ALL: * utf8encode( ) * iconv("ISO-8859-1", "UTF-8//TRANSLIT", $text) * htmlentities( ) * ALTER TABLE '"$TABLE"' CONVERT TO CHARACTER SET utf8 COLLATE utf8_general_ci; * $search = array("\xe2\x80\x9c", "\xe2\x80\x9d", "\xe2\x80\x98", "\xe2\x80\x99"); $replace = array('"', '"', "'", "'"); $outstr = str_replace($search, $replace, $instr) This was REALLY hard. I needed eventually to get out a hex dump to see what those damn characters were. What worked? I did a hex dump (thanks to whoever put up http://www.fileformat.info/tool/hexdump.htm ) and found the mysterious characters. They don't belong to any character convention I know of. They are single-byte versions of some of the UTF-8 two-byte quotes. Single-curly-quotes are 0x91 and 0x92. Double-curly-quotes are 0x93 and 0x94. Ellipses 0x85. So this code fixes it if I don't mind dumb quotes, which believe me, at this point I don't: function dumquote ($instr) { $search = array("\x93", "\x94", "\x91", "\x92", "\x85"); $replace = array('"', '"', "'", "'", "..."); return str_replace($search, $replace, $instr); } Don't forget to escape the resulting dumb quotes afterward with mysql_real_escape_string( ). No clue what character set these come from, but they're listed as the second byte in this UCS-2 table: http://stuffofinterest.com/misc/utf8-about.html ---- Server IP: 69.147.83.197 Probable Submitter: 94.193.241.181 ---- Manual Page -- http://www.php.net/manual/en/function.iconv.php Edit -- https://master.php.net/note/edit/103347 Del: integrated -- https://master.php.net/note/delete/103347/integrated Del: useless -- https://master.php.net/note/delete/103347/useless Del: bad code -- https://master.php.net/note/delete/103347/bad+code Del: spam -- https://master.php.net/note/delete/103347/spam Del: non-english -- https://master.php.net/note/delete/103347/non-english Del: in docs -- https://master.php.net/note/delete/103347/in+docs Del: other reasons-- https://master.php.net/note/delete/103347 Reject -- https://master.php.net/note/reject/103347 Search -- https://master.php.net/manage/user-notes.php