Bug #71444 [Fbk->Opn]: Fatal termination of data length occurs in transforming from EBCDIC to UTF-8

Edit report at https://bugs.php.net/bug.php?id=71444&edit=1 ID: 71444 User updated by: ta_nakagawa at ysco dot net Reported by: ta_nakagawa at ysco dot net Summary: Fatal termination of data length occurs in transforming from EBCDIC to UTF-8 -Status: Feedback +Status: Open Type: Bug Package: ODBC related Operating System: Windows 7 / Windows 2012 R2 PHP Version: 5.6.17 Block user comment: N Private report: N New Comment: Thank you for your reply. A data of EBCDIC multibyte chacacter is needed to insert to the column. We prepared the binary image containing the data. http://www.ysco.net/phpbug/tokmsp_saved You can make the sample data for AS/400 system as below. 1. DDL of the table creation for AS/400 system. CREATE TABLE TOKMSP(TKBANG CHAR (5), TKNAKJ CHAR (20) CCSID 1399) 2. Setup the sample data to AS/400 system in the following stes. The binary image file "tokmsp_saved" contains the sample data. 1) Execute AS/400 command "CRTSAVF QGPL/SAVF". 2) Execute FTP put command by binary image to AS/400 system as below. "binary" "put tokmsp_saved QGPL/SAVF" 3) Execute AS/400 command "RSTOBJ OBJ(TOKMSP) SAVLIB(QGPL) DEV(*SAVF) SAVF(QGPL/SAVF)". Thanks. Previous Comments: ------------------------------------------------------------------------ [2016-02-23 14:04:07] ab@php.net Thanks for the report. Could you please extend your code with the table creation and the relevant data? Thanks. ------------------------------------------------------------------------ [2016-01-25 01:43:10] ta_nakagawa at ysco dot net Description: ------------ When the query returns the UTF-8 multibyte character (such as Japanese characters) from a AS/400 EBCDIC data field via ODBC, the last some characters is terminated in the ODBC module of PHP. The conversion from EBCDIC to UTF-8 increases the size of the characters. The data size of UTF-8 becomes 2.5 times in the case of the size maximizing. ODBC driver (DB2CLI or IBM i access for windows) converts to the UTF-8 characters correctly, but the PHP does not have the buffer storing the incremental size for the conversion. As a result the last some characters is dropped. For example, in AS/400 the data field by 20 bytes can store 9 EBCDIC double byte characters (20 bytes) such as Japanese character. It becomes 27 bytes in UTF-8. When PHP receives the data from the driver, the size of the variable assigned the data is not sufficient. So the last 3 characters corresponding to 7 bytes are lost. This problem can be avoid by extra allocation to the variable of the characters within the function "odbc_bindcols" in the source "\ext\odbc\php_odbc.c" It needs to remove the conditional statement for extra allocation of character. We hope that the problem in an appropriate manner is resolved. Thank you for your consideration. diff -u ext/odbc/php_odbc.c.orig ext/odbc/php_odbc.c --- ext/odbc/php_odbc.c.orig 2016-01-06 07:14:48.000000000 +0900 +++ ext/odbc/php_odbc.c 2016-01-21 22:16:59.767514900 +0900 @@ -1020,10 +1020,9 @@ displaysize += 3; } - if (charextraalloc) { - /* Since we don't know the exact # of bytes, allocate extra */ - displaysize *= 4; - } + /* Since we don't know the exact # of bytes, allocate extra */ + displaysize *= 4; + result->values[i].value = (char *)emalloc(displaysize + 1); rc = SQLBindCol(result->stmt, (SQLUSMALLINT)(i+1), SQL_C_CHAR, result->values[i].value, displaysize + 1, &result->values[i].vallen); Test script: --------------- $conn = odbc_connect("MyDSN", "MyUser", "MyPWD"); $query = 'SELECT TKBANG,TKNAKJ FROM TOKMSP'; $stmt = odbc_prepare($conn, $query); $ret = odbc_execute($stmt); while($record = odbc_fetch_array($ret) ) print_r($record); odbc_close($conn); Expected result: ---------------- Array ( [TKBANG] => 00121 [TKNAKJ] => たちつてとなにぬね ) The character of [TKNAKJ] is Japanese. The data has already been inserted to the column. Actual result: -------------- Array ( [TKBANG] => 00121 [TKNAKJ] => たちつてとな ) The last 3 characters are terminated. ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=71444&edit=1