note 99813 deleted from function.mb-check-encoding by danbrown

Note Submitter: php at linuxhosted dot ca ---- Here is the ultimate UTF-8 validity check function, it is by far the fastest function of any others listed here, and it is one of the only ones that actually pass all validity checks, and does not crash php or hit the recursion limit for PCRE. On an error it will return NULL, otherwise true if $string is valid UTF-8, otherwise false. The common w3 PCRE regular expression match from: http://w3.org/International/questions/qa-forms-utf-8.html does not match all valid UTF-8 in the 0-127 ASCII range, and will also cause a recursion limit or PHP segfault if the input string is too long. <?php function is_utf8($string) { if (!is_string($string)) return NULL; $valid = preg_match('##Dsu', $string); if ($valid === false) { $error = preg_last_error(); switch ($error) { case PREG_BAD_UTF8_ERROR: case PREG_BAD_UTF8_OFFSET_ERROR: return false; default: return NULL; } } return (bool)$valid; } ?> I made a testing script to check the various functions for testing UTF-8 validity for speed and accuracy, the modified w3 preg_match is simply fixed to allow all ascii characters. For the full code of my UTF-8 function testing script, see: http://svn.sctbdev.ca/wsvn/tests/utf8.php Results from my testing script: Building string of all possible valid UTF-8 characters ... DONE in 1.72670s (4382592 bytes) Building random data string of same length ............... DONE in 1.38229s (4382592 bytes) ------------------------------------------------------------------ is_utf8(): Passed validation of all UTF-8 conditions. Testing 64 iterations with Random Data ... PASSED after 0.00005s Testing 64 iterations with Valid Data .... PASSED after 0.51578s ------------------------------------------------------------------ mb_check_encoding(): Valid 4 Byte Sequence (but above Unicode codepoints): FAILED Valid 5 Byte Sequence (but above Unicode codepoints): FAILED Valid 6 Byte Sequence (but above Unicode codepoints): FAILED Failed to validate all UTF-8 conditions correctly. Testing 64 iterations with Random Data ... PASSED after 3.53275s Testing 64 iterations with Valid Data .... PASSED after 3.15102s ------------------------------------------------------------------ mb_detect_encoding(): Valid 2 Byte Sequence (but contains Overlong): FAILED Valid 3 Byte Sequence (but contains Overlong): FAILED Valid 4 Byte Sequence (but contains Overlong): FAILED Valid 5 Byte Sequence (but contains Overlong): FAILED Valid 6 Byte Sequence (but contains Overlong): FAILED Valid 3 Byte Sequence (but contains Surrogate): FAILED Valid 4 Byte Sequence (but above Unicode codepoints): FAILED Valid 5 Byte Sequence (but above Unicode codepoints): FAILED Valid 6 Byte Sequence (but above Unicode codepoints): FAILED Failed to validate all UTF-8 conditions correctly. Testing 64 iterations with Random Data ... PASSED after 1.40612s Testing 64 iterations with Valid Data .... PASSED after 3.09515s ------------------------------------------------------------------ mb_convert_encoding(): Passed validation of all UTF-8 conditions. Testing 64 iterations with Random Data ... PASSED after 6.20198s Testing 64 iterations with Valid Data .... PASSED after 5.36951s ------------------------------------------------------------------ w3 preg_match(): Valid 1 Byte Sequence: FAILED Failed to validate all UTF-8 conditions correctly. Testing 64 iterations with Random Data ... PASSED after 0.00014s Testing 64 iterations with Valid Data .... FAILED after 0.00014s ------------------------------------------------------------------ modified w3 preg_match(): Passed validation of all UTF-8 conditions. Testing 64 iterations with Random Data ... PASSED after 0.00014s Testing 64 iterations with Valid Data .... FAILED after 0.05054s ------------------------------------------------------------------