Bug #71444 [Opn->Fbk]: Fatal termination of data length occurs in transforming from EBCDIC to UTF-8

Edit report at https://bugs.php.net/bug.php?id=71444&edit=1 ID: 71444 Updated by: ab@php.net Reported by: ta_nakagawa at ysco dot net Summary: Fatal termination of data length occurs in transforming from EBCDIC to UTF-8 -Status: Open +Status: Feedback Type: Bug Package: ODBC related Operating System: Windows 7 / Windows 2012 R2 PHP Version: 5.6.17 Block user comment: N Private report: N New Comment: Thanks for the report. Could you please extend your code with the table creation and the relevant data? Thanks. Previous Comments: ------------------------------------------------------------------------ [2016-01-25 01:43:10] ta_nakagawa at ysco dot net Description: ------------ When the query returns the UTF-8 multibyte character (such as Japanese characters) from a AS/400 EBCDIC data field via ODBC, the last some characters is terminated in the ODBC module of PHP. The conversion from EBCDIC to UTF-8 increases the size of the characters. The data size of UTF-8 becomes 2.5 times in the case of the size maximizing. ODBC driver (DB2CLI or IBM i access for windows) converts to the UTF-8 characters correctly, but the PHP does not have the buffer storing the incremental size for the conversion. As a result the last some characters is dropped. For example, in AS/400 the data field by 20 bytes can store 9 EBCDIC double byte characters (20 bytes) such as Japanese character. It becomes 27 bytes in UTF-8. When PHP receives the data from the driver, the size of the variable assigned the data is not sufficient. So the last 3 characters corresponding to 7 bytes are lost. This problem can be avoid by extra allocation to the variable of the characters within the function "odbc_bindcols" in the source "\ext\odbc\php_odbc.c" It needs to remove the conditional statement for extra allocation of character. We hope that the problem in an appropriate manner is resolved. Thank you for your consideration. diff -u ext/odbc/php_odbc.c.orig ext/odbc/php_odbc.c --- ext/odbc/php_odbc.c.orig 2016-01-06 07:14:48.000000000 +0900 +++ ext/odbc/php_odbc.c 2016-01-21 22:16:59.767514900 +0900 @@ -1020,10 +1020,9 @@ displaysize += 3; } - if (charextraalloc) { - /* Since we don't know the exact # of bytes, allocate extra */ - displaysize *= 4; - } + /* Since we don't know the exact # of bytes, allocate extra */ + displaysize *= 4; + result->values[i].value = (char *)emalloc(displaysize + 1); rc = SQLBindCol(result->stmt, (SQLUSMALLINT)(i+1), SQL_C_CHAR, result->values[i].value, displaysize + 1, &result->values[i].vallen); Test script: --------------- $conn = odbc_connect("MyDSN", "MyUser", "MyPWD"); $query = 'SELECT TKBANG,TKNAKJ FROM TOKMSP'; $stmt = odbc_prepare($conn, $query); $ret = odbc_execute($stmt); while($record = odbc_fetch_array($ret) ) print_r($record); odbc_close($conn); Expected result: ---------------- Array ( [TKBANG] => 00121 [TKNAKJ] => たちつてとなにぬね ) The character of [TKNAKJ] is Japanese. The data has already been inserted to the column. Actual result: -------------- Array ( [TKBANG] => 00121 [TKNAKJ] => たちつてとな ) The last 3 characters are terminated. ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=71444&edit=1