Edit report at https://bugs.php.net/bug.php?id=72330&edit=1
ID: 72330
Updated by: cmb@php.net
Reported by: cronfy at gmail dot com
Summary: CSV fields incorrectly split if escape char followed
by UTF chars
-Status: Feedback
+Status: Re-Opened
Type: Bug
Package: Strings related
Operating System: Linux Mint 17.1 Rebecca
PHP Version: Irrelevant
Assigned To: cmb
Block user comment: N
Private report: N
New Comment:
Thanks for further investigating!
> Set locale to 'en_US' or any locale without multi-byte
> characters. This works fine, but I don't think it's a good way.
It appears to be a viable workaround, as long as you are dealing
with valid UTF-8, and the delimiter and enclosing characters are
single-byte.
A cleaner, but less efficient solution would be to convert the
string to a supported encoding, so str_getcsv() works as expected,
and to convert the resulting array elements back to UTF-8.
Either way, the general caveat regarding setlocale() in
multi-threaded environments[1] applies.
So the best solution might be to use a userland implementation
which does not rely on the locale at all, but rather supports
specifying the character encoding of the CSV input. Adding
something like this to PHP would require the RFC process[2].
> In a word, this function is not binary-safe.
In PHP context "binary-safe" usually means that a function
correctly processes strings containing NUL bytes. str_getcsv()
does this[3], so the function is "binary-safe".
Obviously, you are referring to another defintion of
"binary-safe"[4]. In this sense, str_getcsv() is not
"binary-safe", and actually it can't be, because there are
character encodings where the binary representation of the
delimiter and escape character may be *part* of the binary
representation of other characters.
I'm going to improve the relevant documentation, and will close
this ticket afterwards, since there's not much else which could be
done, unfortunately.
[1] <http://www.php.net/manual/en/function.setlocale.php#refsect1-function.setlocale-notes>
[2] <https://wiki.php.net/rfc/howto>
[3] <https://3v4l.org/qMBEG>
[4] <https://en.wikipedia.org/wiki/Binary-safe>
Previous Comments:
------------------------------------------------------------------------
[2018-04-10 11:44:14] ganlvtech at qq dot com
Thank you very much.
I've also tested the script on Ubuntu 16.04 with php 7.2.2. Test passed.
So the problem may only be reproduced on Windows platform with code page set to a code page with
multi-byte chars (e.g. Chinese Simplified cp936).
After searching on the web for half an hour, I found that 'setlocale to utf8 on Windows'
is imposible.
There might be a hack for the problem. Set locale to 'en_US' or any locale without
multi-byte characters. This works fine, but I don't think it's a good way.
In a word, this function is not binary-safe.
------------------------------------------------------------------------
[2018-04-10 10:56:20] cmb@php.net
The given test script works for me as expected (PHP 7.2.2 on a
German Windows 10). I guess there are locale related issues in
your case, since fgetcsv() takes into account LC_CTYPE[1]. Try to
set an appropriate UTF-8 locale[2] before calling str_getcsv().
[1] <http://www.php.net/manual/en/function.fgetcsv.php#refsect1-function.fgetcsv-notes>
[2] <http://www.php.net/manual/en/function.setlocale.php>
------------------------------------------------------------------------
[2018-04-09 18:21:44] ganlvtech at qq dot com
Related To: Bug #72330
------------------------------------------------------------------------
[2018-04-09 18:21:41] ganlvtech at qq dot com
str_getcsv not correctly work with qouted multibyte character
PHP version: 7.2.2
Operating system: Windows 10 zh-CN
Description:
------------
str_getcsv not correctly work with qouted multibyte characters.
When the multibyte characters are simply seperated by comma, everything seems ok.
If the value contains a quotation mark("), I need to escape quotation mark by doubled quotation
mark(""), and quote the value with a pair of quotation mark. And when I try to decode the
csv string by str_getcsv, this value will combined with next value (I lost a column and got two
value together in one column).
There is not just one type of wrong result. But I think every type of wrong result be caused by the
escaped quotation mark.
Bug #72330: CSV fields incorrectly split if escape char followed by UTF chars
Test script:
---------------
<?php
// Test 1
$data = [
"\xE4\xBD\xA0\xE5\xA5\xBD", // ä½ å¥½
"\xE4\xB8\x96\xE7\x95\x8C", // ä¸ç
];
$encoded = implode(',', array_map(function ($value) {
return '"' . str_replace('"', '""', $value) .
'"';
}, $data));
var_dump(str_getcsv($encoded) === $data);
// Test 2
$data = [
"\"\xE5\x95\x8A", // "å
];
$encoded = str_putcsv($data);
var_dump(str_getcsv($encoded) === $data);
/** @link https://bugs.php.net/bug.php?id=64183
*/
function str_putcsv($fields, $delimiter = ',', $enclosure = '"',
$escape_char = '\\') {
$stream = fopen('php://memory', 'w+');
fputcsv($stream, $fields, $delimiter, $enclosure, $escape_char);
rewind($stream);
return stream_get_contents($stream);
}
Expected result:
----------------
bool(true)
bool(true)
Actual result:
--------------
bool(false)
bool(false)
------------------------------------------------------------------------
[2016-10-17 10:10:46] bwoebi@php.net
Automatic comment on behalf of cmb
Revision: http://git.php.net/?p=php-src.git;a=commit;h=f2c2a4be9e466f14677089efe33e20ca0b146809
Log: Fix #72330: CSV fields incorrectly split if escape char followed by UTF chars
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=72330
--
Edit this bug report at https://bugs.php.net/bug.php?id=72330&edit=1