note 100478 added to function.utf8-decode

Got around to some very basic testing of my modified UTF-8 to HTML Entities function. Apparently preg_replace doesn't like null-characters in its patterns. That was the only glaring error (actually a warning, but it caused the returned string to be empty) I've encountered. That's been fixed. Also, support for four-byte UTF-8 sequences has been added. Finally, arithmetic operators have been replaced with bitwise operators, which should make the conversion slightly faster and less prone to wild results given invalid input — even though the invalid input should have been filtered out already. <?PHP function UTF8ToEntities ($string) { /* note: apply htmlspecialchars if desired /before/ applying this function /* Only do the slow convert if there are 8-bit characters */ /* avoid using 0xA0 (\240) in ereg ranges. RH73 does not like that */ if (! ereg("[\200-\237]", $string) and ! ereg("[\241-\377]", $string)) return $string; // reject too-short sequences $string = preg_replace("/[\302-\375]([\001-\177])/", "&#65533;\\1", $string); $string = preg_replace("/[\340-\375].([\001-\177])/", "&#65533;\\1", $string); $string = preg_replace("/[\360-\375]..([\001-\177])/", "&#65533;\\1", $string); $string = preg_replace("/[\370-\375]...([\001-\177])/", "&#65533;\\1", $string); $string = preg_replace("/[\374-\375]....([\001-\177])/", "&#65533;\\1", $string); // reject illegal bytes & sequences // 2-byte characters in ASCII range $string = preg_replace("/[\300-\301]./", "&#65533;", $string); // 4-byte illegal codepoints (RFC 3629) $string = preg_replace("/\364[\220-\277]../", "&#65533;", $string); // 4-byte illegal codepoints (RFC 3629) $string = preg_replace("/[\365-\367].../", "&#65533;", $string); // 5-byte illegal codepoints (RFC 3629) $string = preg_replace("/[\370-\373]..../", "&#65533;", $string); // 6-byte illegal codepoints (RFC 3629) $string = preg_replace("/[\374-\375]...../", "&#65533;", $string); // undefined bytes $string = preg_replace("/[\376-\377]/", "&#65533;", $string); // reject consecutive start-bytes $string = preg_replace("/[\302-\364]{2,}/", "&#65533;", $string); // decode four byte unicode characters $string = preg_replace( "/([\360-\364])([\200-\277])([\200-\277])([\200-\277])/e", "'&#'.((ord('\\1')&7)<<18 | (ord('\\2')&63)<<12 |" . " (ord('\\3')&63)<<6 | (ord('\\4')&63)).';'", $string); // decode three byte unicode characters $string = preg_replace("/([\340-\357])([\200-\277])([\200-\277])/e", "'&#'.((ord('\\1')&15)<<12 | (ord('\\2')&63)<<6 | (ord('\\3')&63)).';'", $string); // decode two byte unicode characters $string = preg_replace("/([\300-\337])([\200-\277])/e", "'&#'.((ord('\\1')&31)<<6 | (ord('\\2')&63)).';'", $string); // reject leftover continuation bytes $string = preg_replace("/[\200-\277]/", "&#65533;", $string); return $string; } ?> Warning: only a bit of accidental testing was done with invalid UTF-8 input, and the result wasn't closely analyzed. But valid UTF-8 doesn't seem to be mangled at all. ---- Server IP: 69.147.83.197 Probable Submitter: 99.177.172.109 ---- Manual Page -- http://www.php.net/manual/en/function.utf8-decode.php Edit -- https://master.php.net/note/edit/100478 Del: integrated -- https://master.php.net/note/delete/100478/integrated Del: useless -- https://master.php.net/note/delete/100478/useless Del: bad code -- https://master.php.net/note/delete/100478/bad+code Del: spam -- https://master.php.net/note/delete/100478/spam Del: non-english -- https://master.php.net/note/delete/100478/non-english Del: in docs -- https://master.php.net/note/delete/100478/in+docs Del: other reasons-- https://master.php.net/note/delete/100478 Reject -- https://master.php.net/note/reject/100478 Search -- https://master.php.net/manage/user-notes.php