[php-src] Issue #8281: mb_convert_encoding "\" (backslash) and "~" (tilde) convert failed to Shift_JIS

Issue: https://github.com/php/php-src/issues/8281 Comment Author: alexdowad > Shift_JIS is still used in active use, and a common use case is importing CSV into Excel. Since > Excel could only import CSV with Shift_JIS for a long time, there may still be cases where it is > converted with SJIS. > > In other words, mb_convert_encoding is done when downloading and uploading CSV. OK. So you have users uploading CSVs which are Shift-JIS encoded, and you also export Shift-JIS-encoded CSVs for download? Are there reasons why you typically need to convert the text in those CSVs to Unicode? Is it correct to say that when your users upload Shift-JIS-encoded CSVs, those may include 0x5C bytes, and the users expect those to be treated as backslashes? How about when you export Shift-JIS-encoded CSVs for download? Is it typical that you need to include halfwidth backslashes in them? Do you offer options for the encoding of these files? Or you export in Shift-JIS encoding by default, because that is what works for the greatest number of your users? > Also, Windows uses \0x5C for the directory separator. If convert this, it will not function as > a directory separator. That's true; but remembering the context here, we are looking at use cases for conversion between Shift-JIS and Unicode, and trying to determine whether there are more cases where interpreting 0x5C/0x7E according to spec is preferable, or more cases where interpreting them as ASCII is preferable. The discussion is not general, but specific to mb_convert_encoding and the needs of its users. Does Shift-JIS text that PHP-based applications receive from users typically include Windows pathnames? After converting to ASCII/UTF-8/etc., would your PHP code typically take those pathnames and interpolate them into a Windows shell command, or pass them to fopen? If so, is there a reason to convert to Unicode, or could the 'raw' Shift-JIS text be used? > This conversion of \0x5C and \0x7E can be confusing. In most Japanese character code > implementations, it seems customary to convert \0x5C and \0x7E untouched. It is certainly confusing. My preference is to follow published specifications when possible; I believe this tends to reduce confusion in the long term. However, if there are strong practical reasons to deviate from specifications, that can certainly be done. However, we do **not** want to flip-flop back and forth. To avoid flip-flopping, we need to thoroughly understand all the implications of either following the spec or deviating from it. After all factors are considered, and as many interested parties as possible are consulted, if the final decision is to change, then we should document the reason for the decision and stick to it. One of the great challenges involved in working on open-source, and especially popular projects like PHP, is that users only speak up when something is not working well for them. With proprietary, in-house software, you generally know who all the users are and can survey them to see what they think about proposed changes. With open source, you often only discover later how your users were impacted by some change. This is a good reason to carefully think changes through and gather as much information as possible before deciding. You mentioned that it seems customary to treat SJIS 0x5C and 0x7E as ASCII; if you can share as many specific examples as possible of existing software which does or doesn't do this, that would be appreciated. What is 'customary' is definitely one important factor to consider, since it shapes people's expectations.