#24309 [Asn->Csd]: mb_detect_encoding return EUC-JP for invalid EUC-JP char sequence

ID: 24309 Updated by: hirokawa@php.net Reported By: jc at mega-bucks dot co dot jp -Status: Assigned +Status: Closed Bug Type: mbstring related Operating System: Linux PHP Version: 4.3.3RC1 Assigned To: hirokawa New Comment: This bug has been fixed in CVS. In case this was a PHP problem, snapshots of the sources are packaged every three hours; this change will be in the next snapshot. You can grab the snapshot at http://snaps.php.net/. In case this was a documentation problem, the fix will show up soon at http://www.php.net/manual/. In case this was a PHP.net website problem, the change will show up on the PHP.net site and on the mirror sites in short time. Thank you for the report, and for helping us make PHP better. I made a mistake. B7F6 is correct byte code in EUC-JP. But, BA7E is not correct byte code in EUC-JP. So, it is not correct EUC-JP byte sequence. mb_detect_encoding() can choose a best candidate in encoding list, but, it can't detect the corruption. mb_detect_encoding() assumes byte characters are not corrupted and it stops the detection if the number of candidate is only one. I added 'strict detection mode' to detect corrupted string in CVS tree. You should specify TRUE in third argument of mb_detect_encoding to use strict detection mode. <?php $str = urldecode('%B7%F6%BA%7E'); echo mb_detect_encoding($str); // output: 'EUC-JP' echo mb_detect_encoding($str,"ASCII,JIS,UTF-8,EUC-JP,SJIS", TRUE); // output: NULL (strict mode) echo mb_detect_encoding($str,NULL,TRUE); // output: NULL (strict mode, second argument is ommited.) ?> Previous Comments: ------------------------------------------------------------------------ [2003-07-07 06:15:52] sniper@php.net Assigned to the one person who knows what this is about.. :) ------------------------------------------------------------------------ [2003-07-01 21:39:00] jc at mega-bucks dot co dot jp hirokawa wrote: "URL decoded byte sequance of 'search_words=%B7%F6%BA%7E' is B7E6+BA7E, which is correct EUC-JP character sequence." You mean the decoded sequence is B7F6+BA7E, not B7E6+BA7E, right? "B7E6+BA7E [...] is correct EUC-JP byte sequence." Again do you mean that B7F6+BA7E is correct EUC-JP? I don't think it is ... Thanks. ------------------------------------------------------------------------ [2003-06-30 20:01:11] jc at mega-bucks dot co dot jp Are you sure? ^_^ I am not an encoding expert so if you say that it is a valid sequence I believe you but ... I am using postgreSQL as a database and it says that it is not a valid EUC sequence. So either PHP is wrong or the database is wrong :) Here is my test code: echo "Checking $string .......<BR>"; $sql = "select id from products where name like '$string'"; $conn = pg_connect("host=$IP port=5432 dbname=$DB user=postgres"); $res = pg_query($conn, $sql); $err_msg = pg_last_error($conn); if (preg_match("/Invalid EUC_JP character sequence found/", $err_msg)) { echo "NOT VALID<BR>"; } The error message returned by the DB is: pg_query(): Query failed: ERROR: Invalid EUC_JP character sequence found (0xba7e) The output is: Checking &#21927;&#65533; ....... NOT VALID I'll post this to the postgreSQL developer's list also in case it is a bug in postgreSQL. If you are certain that this character sequence is valid can you point me to a ressource I can use to show the postgreSQL team that they have a bug that needs fixing? Thanks! ------------------------------------------------------------------------ [2003-06-30 07:49:30] hirokawa@php.net It is not a bug of mbstring. 0xb7,0xf6,0xba,0x7e is a correct byte seqence of EUC-JP. ------------------------------------------------------------------------ [2003-06-28 09:40:13] hirokawa@php.net URL decoded byte sequance of 'search_words=%B7%F6%BA%7E' is B7E6+BA7E, which is correct EUC-JP character sequence. <?php // sample code $str_euc = sprintf("%c%c%c%c",0xb7,0xf6,0xba,0x7e); echo mb_detect_encoding($str_euc); // output is 'EUC-JP' ?> Encoding detection is not perfect, it may make mistake if the length of string is too short. But, I believe encoding detection of mbstring works fine in this case. B7E6+BA7E is not correct byte sequence of SJIS, UTF-8, ISO2022-JP. It is correct EUC-JP byte sequence. ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at http://bugs.php.net/24309 -- Edit this bug report at http://bugs.php.net/?id=24309&edit=1