Bug #72330 [Com]: CSV fields incorrectly split if escape char followed by UTF chars

Edit report at https://bugs.php.net/bug.php?id=72330&edit=1 ID: 72330 Comment by: ganlvtech at qq dot com Reported by: cronfy at gmail dot com Summary: CSV fields incorrectly split if escape char followed by UTF chars Status: Feedback Type: Bug Package: Strings related Operating System: Linux Mint 17.1 Rebecca PHP Version: Irrelevant Assigned To: cmb Block user comment: N Private report: N New Comment: Thank you very much. I've also tested the script on Ubuntu 16.04 with php 7.2.2. Test passed. So the problem may only be reproduced on Windows platform with code page set to a code page with multi-byte chars (e.g. Chinese Simplified cp936). After searching on the web for half an hour, I found that 'setlocale to utf8 on Windows' is imposible. There might be a hack for the problem. Set locale to 'en_US' or any locale without multi-byte characters. This works fine, but I don't think it's a good way. In a word, this function is not binary-safe. Previous Comments: ------------------------------------------------------------------------ [2018-04-10 10:56:20] cmb@php.net The given test script works for me as expected (PHP 7.2.2 on a German Windows 10). I guess there are locale related issues in your case, since fgetcsv() takes into account LC_CTYPE[1]. Try to set an appropriate UTF-8 locale[2] before calling str_getcsv(). [1] <http://www.php.net/manual/en/function.fgetcsv.php#refsect1-function.fgetcsv-notes> [2] <http://www.php.net/manual/en/function.setlocale.php> ------------------------------------------------------------------------ [2018-04-09 18:21:44] ganlvtech at qq dot com Related To: Bug #72330 ------------------------------------------------------------------------ [2018-04-09 18:21:41] ganlvtech at qq dot com str_getcsv not correctly work with qouted multibyte character PHP version: 7.2.2 Operating system: Windows 10 zh-CN Description: ------------ str_getcsv not correctly work with qouted multibyte characters. When the multibyte characters are simply seperated by comma, everything seems ok. If the value contains a quotation mark("), I need to escape quotation mark by doubled quotation mark(""), and quote the value with a pair of quotation mark. And when I try to decode the csv string by str_getcsv, this value will combined with next value (I lost a column and got two value together in one column). There is not just one type of wrong result. But I think every type of wrong result be caused by the escaped quotation mark. Bug #72330: CSV fields incorrectly split if escape char followed by UTF chars Test script: --------------- <?php // Test 1 $data = [ "\xE4\xBD\xA0\xE5\xA5\xBD", // 你好 "\xE4\xB8\x96\xE7\x95\x8C", // 世界 ]; $encoded = implode(',', array_map(function ($value) { return '"' . str_replace('"', '""', $value) . '"'; }, $data)); var_dump(str_getcsv($encoded) === $data); // Test 2 $data = [ "\"\xE5\x95\x8A", // "啊 ]; $encoded = str_putcsv($data); var_dump(str_getcsv($encoded) === $data); /** @link https://bugs.php.net/bug.php?id=64183 */ function str_putcsv($fields, $delimiter = ',', $enclosure = '"', $escape_char = '\\') { $stream = fopen('php://memory', 'w+'); fputcsv($stream, $fields, $delimiter, $enclosure, $escape_char); rewind($stream); return stream_get_contents($stream); } Expected result: ---------------- bool(true) bool(true) Actual result: -------------- bool(false) bool(false) ------------------------------------------------------------------------ [2016-10-17 10:10:46] bwoebi@php.net Automatic comment on behalf of cmb Revision: http://git.php.net/?p=php-src.git;a=commit;h=f2c2a4be9e466f14677089efe33e20ca0b146809 Log: Fix #72330: CSV fields incorrectly split if escape char followed by UTF chars ------------------------------------------------------------------------ [2016-07-21 17:13:41] cmb@php.net Automatic comment on behalf of cmb Revision: http://git.php.net/?p=php-src.git;a=commit;h=f2c2a4be9e466f14677089efe33e20ca0b146809 Log: Fix #72330: CSV fields incorrectly split if escape char followed by UTF chars ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=72330 -- Edit this bug report at https://bugs.php.net/bug.php?id=72330&edit=1