Bug #72330 [Com]: CSV fields incorrectly split if escape char followed by UTF chars

Edit report at https://bugs.php.net/bug.php?id=72330&edit=1 ID: 72330 Comment by: ganlvtech at qq dot com Reported by: cronfy at gmail dot com Summary: CSV fields incorrectly split if escape char followed by UTF chars Status: Closed Type: Bug Package: Strings related Operating System: Linux Mint 17.1 Rebecca PHP Version: Irrelevant Assigned To: cmb Block user comment: N Private report: N New Comment: Thanks for your systematical interperation. This might not be a bug, and it can be closed now. Thank you. Previous Comments: ------------------------------------------------------------------------ [2018-04-10 17:00:55] cmb@php.net Automatic comment from SVN on behalf of cmb Revision: http://svn.php.net/viewvc/?view=revision&amp;revision=344648 Log: Clarify locale awareness of the CSV reading functions See bug #72330. ------------------------------------------------------------------------ [2018-04-10 16:48:40] cmb@php.net Thanks for further investigating! > Set locale to 'en_US' or any locale without multi-byte > characters. This works fine, but I don't think it's a good way. It appears to be a viable workaround, as long as you are dealing with valid UTF-8, and the delimiter and enclosing characters are single-byte. A cleaner, but less efficient solution would be to convert the string to a supported encoding, so str_getcsv() works as expected, and to convert the resulting array elements back to UTF-8. Either way, the general caveat regarding setlocale() in multi-threaded environments[1] applies. So the best solution might be to use a userland implementation which does not rely on the locale at all, but rather supports specifying the character encoding of the CSV input. Adding something like this to PHP would require the RFC process[2]. > In a word, this function is not binary-safe. In PHP context "binary-safe" usually means that a function correctly processes strings containing NUL bytes. str_getcsv() does this[3], so the function is "binary-safe". Obviously, you are referring to another defintion of "binary-safe"[4]. In this sense, str_getcsv() is not "binary-safe", and actually it can't be, because there are character encodings where the binary representation of the delimiter and escape character may be *part* of the binary representation of other characters. I'm going to improve the relevant documentation, and will close this ticket afterwards, since there's not much else which could be done, unfortunately. [1] <http://www.php.net/manual/en/function.setlocale.php#refsect1-function.setlocale-notes> [2] <https://wiki.php.net/rfc/howto> [3] <https://3v4l.org/qMBEG> [4] <https://en.wikipedia.org/wiki/Binary-safe> ------------------------------------------------------------------------ [2018-04-10 11:44:14] ganlvtech at qq dot com Thank you very much. I've also tested the script on Ubuntu 16.04 with php 7.2.2. Test passed. So the problem may only be reproduced on Windows platform with code page set to a code page with multi-byte chars (e.g. Chinese Simplified cp936). After searching on the web for half an hour, I found that 'setlocale to utf8 on Windows' is imposible. There might be a hack for the problem. Set locale to 'en_US' or any locale without multi-byte characters. This works fine, but I don't think it's a good way. In a word, this function is not binary-safe. ------------------------------------------------------------------------ [2018-04-10 10:56:20] cmb@php.net The given test script works for me as expected (PHP 7.2.2 on a German Windows 10). I guess there are locale related issues in your case, since fgetcsv() takes into account LC_CTYPE[1]. Try to set an appropriate UTF-8 locale[2] before calling str_getcsv(). [1] <http://www.php.net/manual/en/function.fgetcsv.php#refsect1-function.fgetcsv-notes> [2] <http://www.php.net/manual/en/function.setlocale.php> ------------------------------------------------------------------------ [2018-04-09 18:21:44] ganlvtech at qq dot com Related To: Bug #72330 ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=72330 -- Edit this bug report at https://bugs.php.net/bug.php?id=72330&edit=1