Req #73716 [Com]: PHP 7.1's CHCP switching in console should be optional

Edit report at https://bugs.php.net/bug.php?id=73716&edit=1 ID: 73716 Comment by: anrdaemon at freemail dot ru Reported by: anrdaemon at freemail dot ru Summary: PHP 7.1's CHCP switching in console should be optional Status: Open Type: Feature/Change Request Package: Output Control Operating System: Windows PHP Version: 7.1.0 Block user comment: N Private report: N New Comment: Said that, let's take *NIX as example. When you are writing to terminal in *NIX, you don't suddenly change terminal codepage, you translate your data from your program's internal codepage to the terminal's one. Why on earth Windows terminal has to be any different? I can imagine the time it took for you to write all that text, but it's senseless. How's "UTF-8 is undesirable"? It IS desirable. Internally. I want my application to use UTF-8 wherever possible. Emphasis on "possible". As opposed to "wherever it want regardless of my expectations". I'm reading your argumentations and all my reaction is an urge to shake my head in an attempt to get the wrongs out of it. Previous Comments: ------------------------------------------------------------------------ [2016-12-13 05:04:08] anrdaemon at freemail dot ru World doesn't rotate around PHP, and there's other programs writing to the same console at the same time. Which totally do not expect random CP changes. Least of all, I totally do not expect CP changes from INTERNAL program settings. ------------------------------------------------------------------------ [2016-12-12 21:30:39] ab@php.net A bit more background regarding these behaviors. The world is UTF-8 today. The Windows path issue, both long and UTF-8, was long standing. UTF-8 is now default in PHP on Windows, like it became relatively long ago on Linux and other platforms, and in PHP-5.6 not very long ago. Still Windows is very different from other internationalization approaches, despite some improvements in system locale handling are to see. Many APIs are still codepage bound, including console, path, I/O, etc. That is unlikely to change soon, if at all. This makes the portability of PHP on Windows itself to lag. So for one - the behavior in PHP needs to be more consolidated across platform, for the other - it needs to be done a simple way. But even then - there are various platform issues, so then the actual thing is sometimes tricky to stretch straight. @anrdiemon, the INI configuration listed is inconsistent for 7.1 and even for earlier. For 7.1, it diverges from what was documented in UPGRADING in first place. Then, the output_encoding directive is only useful, if the usage of the iconv/mbstring ob handler use is intended. As Yasuo mentioned, this ob handler has no effect on CLI. Furthermore - any of *_encoding are deprecated, see http://php.net/manual/en/iconv.configuration.php Here's what i have with a non existent extension DLL [code] C:\php-sdk\php71\vc14\x64\php-src $ chcp Active code page: 437 C:\php-sdk\php71\vc14\x64\php-src $ x64\Release\php.exe -n -d extension_dir=nowhere -d extension=notfound -v PHP Warning: PHP Startup: Unable to load dynamic library 'nowhere\notfound' - The specified module could not be found. in Unknown on line 0 PHP 7.1.1-dev (cli) (built: Dec 12 2016 15:15:56) ( NTS MSVC14 (Visual C++ 2015) x64 ) Copyright (c) 1997-2016 The PHP Group Zend Engine v3.1.0, Copyright (c) 1998-2016 Zend Technologies C:\php-sdk\php71\vc14\x64\php-src $ chcp Active code page: 437 [/code] The encoding determination sequence, as specifically documented in UPGRADING, is kept same, despite internal_encoding is used. Otherwise, there is no behavior difference, neither in earlier PHP version, nor on another platform. What is new - yes, the console codepage is switched automatically, but that is not without a reason. The console is UTF-8 on the overwhelming number of platforms. @requinix, that's a good catch. Of course, if a process is sent a KILL, it won't be able to handle it. Same will happen when SIGSEGV and several other situations occur. This kind of behavior is what i expect @anrdaemon experiences. Any controlled exit from will sure restore the console, but if -9 is sent - there's nothing that can be done. So this is a real crash in a C program, that have to be happening. As for me, adding an INI to just workaround a crash, is not sensible. And probably, doing this colud be even misleading. Imagine, you output a cyrillic string with 7.1, while the console codepage is 437 - [code] C:\php-sdk\php71\vc14\x64\php-src $ chcp Active code page: 437 C:\php-sdk\php71\vc14\x64\php-src $ x64\Release\php.exe -n -d default_charset="CP866" -r "var_dump(sapi_windows_cp_get(), 'привет');" int(866) string(6) "привет" C:\php-sdk\php71\vc14\x64\php-src $ chcp Active code page: 437 C:\php-sdk\php71\vc14\x64\php-src $ [/code] If there were default_charset=cp437 directive set to 7.1, what it gave is this - string(6) "??????", just like 7.0 would do by default. With default_charset=UTF-8 as default in 7.0, it were same, but in 7.1, it's [code] string(12) "привет" int(65001) [/code] Now, if one would want to out another one, say a Czech string - this is broken again. The only what works is - using UTF-8 console output. This conserns same for direct input, or for warnings, error messages, open and output paths, getting various data like user names, et cetera. Even there were an INI, and the output would be turned off, either one or the other of the cases will put mojibake onto console. Other side - the input will be in incompatible incoding. If the console is, say cp866, but PHP uses UTF-8 internaly, and you want to read a filename from console to put it into an I/O functionin. PHP will internally try to convert the cp866 char's into wchar_t's using utf-8. Now, we can of course say - lets use different codepage for input, than convert it to internal, then convert again into output. Diverging codepages for PHP internal, PHP input, PHP output, console input, console output ... well, hopefully one can see where it leads. Subsequent weirdness and bug reports! are guaranteed :) The way to do it, if UTF-8 is not desired, is simply setting like internal_encoding=cp1251, which will keep the current console codepage but also disable the multibyte path and other feature support. In this case - all the behavior is turned to what it was before 7.1. This is documented and done this way to explicitly keep the the backward compatibility for older apps or for scripts requiring the old behavior. Sure, any implementation can't be perfect enough, this one exhausts the most of the possibilities systems provide. If interested, it were also worth it to check, how this topic is handled in other language, Python for example :) @anrdaemon, the reported issue is something, that is caused by not following the recommended UPGRADING way. The ini configuration is not supposed to deliver the expected result in any case. With the crash behavior as described by @requinix- yeah, that's a known one, but it is something different. It is noticeable in an abnormal crash situation and is impossible to catch. Any other program won't behave different in this case. Now, if we say, PHP crashes that often, that it becomes an issue of this kind - then we have something else to fix. UTF-8 and the wide APIs usage is what really matters for the future and is the worthy goal to strive. The old PHP on Windows behavior is still available by putting the corresponding configuration. UTF-8 became default in 5.6, now it's reflected on the Windows side as well. There is a number of factors, that can make UTF-8 usage not as easy as on other platforms. There are Windows specific things, and there is some learn curve, but there is no reason to mix the old and new behavior. Either an app has a clean UTF-8 support, or it goes by the legacy behavior. A "half UTF-8" support an over complicated implementation would be something weird, as for me. If there's a crash scenario to investigate, that'd what should be done. But otherwise, i'd see it as "not a bug", same as Yasuo. Thanks. ------------------------------------------------------------------------ [2016-12-12 02:56:16] requinix@php.net Reproduce script: (Windows 10, en_US) >chcp Active code page: 437 >php -r "system('taskkill /f /pid '.getmypid());" >chcp Active code page: 65001 I wouldn't call this a PHP bug because the "normal" exit situations I tried, including fatal errors, would revert the codepage correctly. The problem comes when the PHP process itself quits, so PHP doesn't have a chance to recover and that could cause other issues on its own. Setting the codepage to match the internal_encoding is a net benefit. That said, I think this could be controlled by an INI setting, default enabled. The normal reasons not to add one don't apply here: for example, the value of the setting does not impact code so PHP libraries don't need to detect the setting's value to alter their behavior. If disabled then code still works the same way, but output may be mojibake-d. And if the output is redirected to a file then the setting doesn't even matter at all. So I'm going to convert this to a feature request. ------------------------------------------------------------------------ [2016-12-12 00:37:24] yohgaki@php.net BTW, I don't think mbstring converts console input. You might want to create feature request for this. ------------------------------------------------------------------------ [2016-12-12 00:34:54] yohgaki@php.net If you would like to convert input/output encoding, you need to use mbstring/iconv module's conversion feature. This isn't enabled by default and you should enabled them by yourself. ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=73716 -- Edit this bug report at https://bugs.php.net/bug.php?id=73716&edit=1