note 100477 added to function.utf8-decode

I've extended Blackbit's example below to reject bytes and sequences that are illegal in UTF-8. They are replaced with the replacement character '�' (U+FFFD). <?PHP function charset_decode_utf_8 ($string) { /* note: apply html_entities if desired /before/ applying this function /* Only do the slow convert if there are 8-bit characters */ /* avoid using 0xA0 (\240) in ereg ranges. RH73 does not like that */ if (! ereg("[\200-\237]", $string) and ! ereg("[\241-\377]", $string)) return $string; // reject too-short sequences $string = preg_replace("/[\302-\375]([\000-\177])/", "&#65533;\\1", $string); $string = preg_replace("/[\340-\375].([\000-\177])/", "&#65533;\\1", $string); $string = preg_replace("/[\360-\375]..([\000-\177])/", "&#65533;\\1", $string); $string = preg_replace("/[\370-\375]...([\000-\177])/", "&#65533;\\1", $string); $string = preg_replace("/[\374-\375]....([\000-\177])/", "&#65533;\\1", $string); // reject illegal bytes & sequences $string = preg_replace("/[\300-\301]./", "&#65533;", $string); // 2-byte characters in ASCII range $string = preg_replace("/\364[\220-\277]../", "&#65533;", $string); // 4-byte illegal codepoints (RFC 3629) $string = preg_replace("/[\365-\367].../", "&#65533;", $string); // 4-byte illegal codepoints (RFC 3629) $string = preg_replace("/[\370-\373]..../", "&#65533;", $string); // 5-byte illegal codepoints (RFC 3629) $string = preg_replace("/[\374-\375]...../", "&#65533;", $string); // 6-byte illegal codepoints (RFC 3629) $string = preg_replace("/[\376-\377]/", "&#65533;", $string); // undefined bytes // reject consecutive start-bytes $string = preg_replace("/[\302-\364]{2,}/", "&#65533;", $string); // decode three byte unicode characters $string = preg_replace("/([\340-\357])([\200-\277])([\200-\277])/e", "'&#'.((ord('\\1')-224)*4096 + (ord('\\2')-128)*64 + (ord('\\3')-128)).';'", $string); // decode two byte unicode characters $string = preg_replace("/([\300-\337])([\200-\277])/e", "'&#'.((ord('\\1')-192)*64+(ord('\\2')-128)).';'", $string); // reject leftover continuation bytes $string = preg_replace("/[\200-\277]/", "&#65533;", $string); return $string; } ?> By rejecting illegal sequences, the chances are greatly reduced of accidentally producing special characters that weren't already clearly in the string. Warning: I haven't tested this yet. ---- Server IP: 69.147.83.197 Probable Submitter: 99.177.172.109 ---- Manual Page -- http://www.php.net/manual/en/function.utf8-decode.php Edit -- https://master.php.net/note/edit/100477 Del: integrated -- https://master.php.net/note/delete/100477/integrated Del: useless -- https://master.php.net/note/delete/100477/useless Del: bad code -- https://master.php.net/note/delete/100477/bad+code Del: spam -- https://master.php.net/note/delete/100477/spam Del: non-english -- https://master.php.net/note/delete/100477/non-english Del: in docs -- https://master.php.net/note/delete/100477/in+docs Del: other reasons-- https://master.php.net/note/delete/100477 Reject -- https://master.php.net/note/reject/100477 Search -- https://master.php.net/manage/user-notes.php