Bug #71444 [Opn]: Fatal termination of data length occurs in transforming from EBCDIC to UTF-8

Edit report at https://bugs.php.net/bug.php?id=71444&edit=1 ID: 71444 User updated by: ta_nakagawa at ysco dot net Reported by: ta_nakagawa at ysco dot net Summary: Fatal termination of data length occurs in transforming from EBCDIC to UTF-8 Status: Open Type: Bug Package: ODBC related Operating System: Windows 7 / Windows 2012 R2 PHP Version: 5.6.17 Block user comment: N Private report: N New Comment: What's the status now? We think the appropriate solution for this problem was selected. When a maintainer is assigned, Could you deal with the problem? Previous Comments: ------------------------------------------------------------------------ [2016-03-12 12:47:16] ta_nakagawa at ysco dot net Thanks for your comment. It is difficult to determine whether the charset of the column of type CHAR is EBCDIC or not. (But A character of charset EBCDIC is usualy inserted a column of type CHAR.) We cannot get the charset information from the driver via ODBC to AS/400. I also think that the fix using the condition with HAVE_IBMDB2 is appropriate. When a user faces a similar situation with our case, a official PHP binary does not solve the problem. It is needed the PHP compiled with the configure option "--with-ibm-db2". Could a php user get a guidance to introduce PHP connecting to DB2, such as described above? ------------------------------------------------------------------------ [2016-03-01 14:34:42] ab@php.net Ups, wrong status ------------------------------------------------------------------------ [2016-03-01 14:34:20] ab@php.net Thanks for the extended information. Unfortunately I have no access to any AS400 machines to test and debug. I hoped to be able to reproduce this with some other drivers like SqlServer, but it seems not passible with another driver with the current snippet because it knows nothing about the EBCDIC charsets :( After some analyze I actually see the issue now from the table creation statement. The column where the EBCDIC data is stored is of type CHAR, and from the specification it doesn't seem to diverge from the specs https://www-01.ibm.com/support/knowledgecenter/SSEPEK_11.0.0/com.ibm.db2z11.doc.odbc/src/tpc/db2z_csql.dita . What i currently don't see and you maybe could point this out - where does the conversion to UTF-8 happen? Does that require some additional options to be done automatically, like when reading the data? As currently, if you look at php_odbc_fetch_hash() - it's always fetched as SQL_C_CHAR, so there should be no conversions to UTF-8 or whatever else. IMO your patch suggestion is not quite appropriate. As you can read from the code, the charextraalloc is exactly for handling the Unicode columns. If the column is not unicode, thus it's a C string and doesn't need any conversion. If we remove this condition, that will mean for any even non multibyte strings the allocation size will be quadrupled. That doesn't sound right. I haven't dig deep in the IBM specifications, but it could be well something DB2 specific. I see a couple of places in the code currently conditioned with HAVE_IBMDB2. IMO the correct fix sohuld recognize the need on extra space and set the charextraalloc = 1; accordingly. Maybe reading the column charset or other driver specific data? Thanks. ------------------------------------------------------------------------ [2016-02-29 01:26:20] ta_nakagawa at ysco dot net Thank you for your reply. A data of EBCDIC multibyte chacacter is needed to insert to the column. We prepared the binary image containing the data. http://www.ysco.net/phpbug/tokmsp_saved You can make the sample data for AS/400 system as below. 1. DDL of the table creation for AS/400 system. CREATE TABLE TOKMSP(TKBANG CHAR (5), TKNAKJ CHAR (20) CCSID 1399) 2. Setup the sample data to AS/400 system in the following stes. The binary image file "tokmsp_saved" contains the sample data. 1) Execute AS/400 command "CRTSAVF QGPL/SAVF". 2) Execute FTP put command by binary image to AS/400 system as below. "binary" "put tokmsp_saved QGPL/SAVF" 3) Execute AS/400 command "RSTOBJ OBJ(TOKMSP) SAVLIB(QGPL) DEV(*SAVF) SAVF(QGPL/SAVF)". Thanks. ------------------------------------------------------------------------ [2016-02-23 14:04:07] ab@php.net Thanks for the report. Could you please extend your code with the table creation and the relevant data? Thanks. ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=71444 -- Edit this bug report at https://bugs.php.net/bug.php?id=71444&edit=1