Bug #71444 [NEW]: Fatal termination of data length occurs in transforming from EBCDIC to UTF-8

From: ta_nakagawa at ysco dot net Operating system: Windows 7 / Windows 2012 R2 PHP version: 5.6.17 Package: ODBC related Bug Type: Bug Bug description:Fatal termination of data length occurs in transforming from EBCDIC to UTF-8 Description: ------------ When the query returns the UTF-8 multibyte character (such as Japanese characters) from a AS/400 EBCDIC data field via ODBC, the last some characters is terminated in the ODBC module of PHP. The conversion from EBCDIC to UTF-8 increases the size of the characters. The data size of UTF-8 becomes 2.5 times in the case of the size maximizing. ODBC driver (DB2CLI or IBM i access for windows) converts to the UTF-8 characters correctly, but the PHP does not have the buffer storing the incremental size for the conversion. As a result the last some characters is dropped. For example, in AS/400 the data field by 20 bytes can store 9 EBCDIC double byte characters (20 bytes) such as Japanese character. It becomes 27 bytes in UTF-8. When PHP receives the data from the driver, the size of the variable assigned the data is not sufficient. So the last 3 characters corresponding to 7 bytes are lost. This problem can be avoid by extra allocation to the variable of the characters within the function "odbc_bindcols" in the source "\ext\odbc\php_odbc.c" It needs to remove the conditional statement for extra allocation of character. We hope that the problem in an appropriate manner is resolved. Thank you for your consideration. diff -u ext/odbc/php_odbc.c.orig ext/odbc/php_odbc.c --- ext/odbc/php_odbc.c.orig 2016-01-06 07:14:48.000000000 +0900 +++ ext/odbc/php_odbc.c 2016-01-21 22:16:59.767514900 +0900 @@ -1020,10 +1020,9 @@ displaysize += 3; } - if (charextraalloc) { - /* Since we don't know the exact # of bytes, allocate extra */ - displaysize *= 4; - } + /* Since we don't know the exact # of bytes, allocate extra */ + displaysize *= 4; + result->values[i].value = (char *)emalloc(displaysize + 1); rc = SQLBindCol(result->stmt, (SQLUSMALLINT)(i+1), SQL_C_CHAR, result->values[i].value, displaysize + 1, &result->values[i].vallen); Test script: --------------- $conn = odbc_connect("MyDSN", "MyUser", "MyPWD"); $query = 'SELECT TKBANG,TKNAKJ FROM TOKMSP'; $stmt = odbc_prepare($conn, $query); $ret = odbc_execute($stmt); while($record = odbc_fetch_array($ret) ) print_r($record); odbc_close($conn); Expected result: ---------------- Array ( [TKBANG] => 00121 [TKNAKJ] => たちつてとなにぬね ) The character of [TKNAKJ] is Japanese. The data has already been inserted to the column. Actual result: -------------- Array ( [TKBANG] => 00121 [TKNAKJ] => たちつてとな ) The last 3 characters are terminated. -- Edit bug report at https://bugs.php.net/bug.php?id=71444&edit=1 -- Try a snapshot (PHP 5.4): https://bugs.php.net/fix.php?id=71444&r=trysnapshot54 Try a snapshot (PHP 5.5): https://bugs.php.net/fix.php?id=71444&r=trysnapshot55 Try a snapshot (trunk): https://bugs.php.net/fix.php?id=71444&r=trysnapshottrunk Fixed in SVN: https://bugs.php.net/fix.php?id=71444&r=fixed Fixed in release: https://bugs.php.net/fix.php?id=71444&r=alreadyfixed Need backtrace: https://bugs.php.net/fix.php?id=71444&r=needtrace Need Reproduce Script: https://bugs.php.net/fix.php?id=71444&r=needscript Try newer version: https://bugs.php.net/fix.php?id=71444&r=oldversion Not developer issue: https://bugs.php.net/fix.php?id=71444&r=support Expected behavior: https://bugs.php.net/fix.php?id=71444&r=notwrong Not enough info: https://bugs.php.net/fix.php?id=71444&r=notenoughinfo Submitted twice: https://bugs.php.net/fix.php?id=71444&r=submittedtwice register_globals: https://bugs.php.net/fix.php?id=71444&r=globals PHP 4 support discontinued: https://bugs.php.net/fix.php?id=71444&r=php4 Daylight Savings: https://bugs.php.net/fix.php?id=71444&r=dst IIS Stability: https://bugs.php.net/fix.php?id=71444&r=isapi Install GNU Sed: https://bugs.php.net/fix.php?id=71444&r=gnused Floating point limitations: https://bugs.php.net/fix.php?id=71444&r=float No Zend Extensions: https://bugs.php.net/fix.php?id=71444&r=nozend MySQL Configuration Error: https://bugs.php.net/fix.php?id=71444&r=mysqlcfg