Bug #74312 [Com]: Filter BOMs out of PHP output

Edit report at https://bugs.php.net/bug.php?id=74312&edit=1 ID: 74312 Comment by: spam2 at rhsoft dot net Reported by: furun at arcor dot de Summary: Filter BOMs out of PHP output Status: Re-Opened Type: Bug Package: Unicode Engine related Operating System: Win7 PHP Version: 7.0.17 Block user comment: N Private report: N New Comment: > and i expect code incompatibility (not BOM) by working on PHP5 > PHP7. > so nothing like "5 min google search" what did you not understand in "that ship sailed long ago"? PHP7.0 is GA since 2015 PHP7.1 is GA since 2016 just because you have not finished "working on PHP5 > PHP7" don't mean you can go ahead and make backward incompatible changes at this point in time - to make it clear: our current deploayable code base of 250000 line sof code is PHP7 only (return types, scalar type hints, declare(strict_types=1);) and in a short will be PHP7.1 only by introduce void-return-types all over the place and that's what "that ship sailed long ago" means Previous Comments: ------------------------------------------------------------------------ [2017-04-01 23:43:26] furun at arcor dot de (nikic) I read the link you post here, tanks. http://markmail.org/message/besjw22hxlpwlvdh Needles to say that i am on the arguments from Sammy Kaye Powers. (Repeating my self now...: I searched a little in files, and found more BOMs in the files then i expected. It was a quick search. Like expected, in plugins mainly, and not the main codes. In JS and CSS files too, but browsers are now all handling it, JS BOMs don't cause problems anymore in browsers. Some projects have a "don't use BOM and ending ?>" advice. And i found in quick search several posts in forums BOM and ?> related, the "headers already sent by" error is maybe most common, but by fare not the only one. The BOM is nasty, because it creates errors all over the place, "wearing an invisibility cloak so-to-speak". In my case it was a broken captsha image generator, because of a 3-party plugin far away. It was a nerving bug search because 2 scripts had a BOM, and i searched a while on the wrong sides. Because i clean this bug source in my codes, i was not trained to search the bug there, and i expect code incompatibility (not BOM) by working on PHP5 > PHP7. So nothing like "5 min google search".! Experienced forum workers maybe know it better, but i think it is a source for recurring nasty issues... and i would fix it like discusses there, in the lexer. Mono develop onces destroy one of my C# files, it was a file to handle UFT-8 encoding problems, and then gets problems it self. I was suddenly simply defect, [?]-Chars, and i never worked on it for long time. Since then, all my C# files have the UTF-8 BOM to tell Mono, "don't destroy it again, it is a UTF-8, leave it so". So against the advice, "do not use UFT-8 BOMs". Link from Sammy Kaye Powers, and so i saw it too. http://stackoverflow.com/search?q=php+bom ) ------------------------------------------------------------------------ [2017-04-01 21:32:40] spam2 at rhsoft dot net > My opinion is, intentional BOM is rare, and the PHP5 > PHP7 > upgrade is the right time to the change, in case it breaks > existing code, wen all developers are busy with maintenance > and compatibility anyway. that ship sailed long ago > If the BOM and maybe even ?> handling is done behind an ini > setting, its even better, it gives developers the choice no it don't - it only makes it unpredictable if, when and where your code breaks since you as developer normally have no control about ini-settings ------------------------------------------------------------------------ [2017-04-01 21:22:48] furun at arcor dot de (nikic) a) A technical argument could be, it could cause other problems if you just strip the BOM, and not process some UTF-8/encoding stuf. php must/should maybe then care some how what the BOM says about the encoding. My question would be then, if php still does not care about file encoding, would it break something/more then leaf BOMs in the files? I think if a BOM is there, php could presume correct and indented encoding. For UTF/encoding handling then is definitely the code responsible and not php, even UTF error corrections. (i use a script for this. Far back, wen some software switched from ASCII to UTF8, i have seen some confused files, and i fixed them by script, instead to manually correct all [?]-characters). I miss there maybe something, but my suggestion would be to just delete/ignore the BOM, and keep the file as it is. (Keep the present behavior in default configuration.) (I never used zend.multibyte, i like if php gives all out exactly as it is. The only exception is BOM and empty chars after ?>, where i see recurring problems.) b) A other technical argument is backwards compatibility. My suggestion is, if a BOM output is intentional, it could be done explicit in code. And it would be a other bug source if a file not have, but should have a BOM for the output. Because the BOM is in the editor likely only visible in the options and not in the text editor it self, so not visible in first view (NotePad++ for example). Explicit BOM export would be more clean. And i would speculate, a intentional export of a BOM is way less probable, then a buggy behavior with unintentional BOMs. And a developer is focused on this specific behavior to export BOMs, other then unintentional BOMs. (PHP could define constants for all important BOMs, for explicit output, if not done already.) The discussion i searched here is the possible backwards compatibility break, and if problems are probable and recurring as unintended BOMs. My opinion is, intentional BOM is rare, and the PHP5 > PHP7 upgrade is the right time to the change, in case it breaks existing code, wen all developers are busy with maintenance and compatibility anyway. If the BOM and maybe even ?> handling is done behind an ini setting, its even better, it gives developers the choice. The question and discussion is then, should it be the default setting to handle or not handle the BOM by php. I would vote with my present opinion, to strip BOM by default in ini setting, in PHP-7, maybe not in PHP-5(?) if backwards compatibility is a big issue. The big question is, is BOM and empty chars after ?> a recurring and potentially nasty issue? In my opinion it is, which could be fixed pragmatically simply by changing some php roles. Correct my if i am wrong there. ------------------------------------------------------------------------ [2017-04-01 17:04:32] spam2 at rhsoft dot net > to reeducate the entire planet yes, because such "education" is a prerequisite anyways - or where you born with knowing programming languages and even if why that one piece was missing > or to take the stone out of the way there is no stone - frankly looking at 90% of PHP code out there (leading to prohibit as much as possible 3rd party code on our sevrers for a decade now) there are way too less stones checking if someone should prdocuece code running on sevrers for which he is most of the time not responsile at his own and innocent sysadmins and attacked 3rd party sites have to chew the outcome later ------------------------------------------------------------------------ [2017-04-01 15:39:10] nikic@php.net Lets not start calling people idiots... Here is a semi-recent internals thread on the topic of BOM handling: http://markmail.org/message/besjw22hxlpwlvdh My opinion here is essentially this: a) First, a philosophical point. PHP, in a default configuration, does not care about file encoding. As long as it's ASCII compatible, PHP does not care whether your file uses UTF-8, ISO-8859-1 or Windows-1251. PHP does not perform any validation or conversion, it gives you everything back exactly as it received it. There *is* a special mode in which PHP does all these things, and that's zend.multibyte. If this mode is enabled, PHP can convert character encodings, strip BOMs, etc. Of course, we know that in practice there is very little interest in this. b) Second, a practical point. You are considering the case where the BOM was accidentally introduced by bad tooling (say, people writing PHP code in MS Word). However, it may also be introduced *intentionally*, to be produced as part of the output. (For the sake of an example, assume you have template files including BOMs, because you want to deliver a response with BOMs, as tooling on the consuming side requires it.) If nothing else, this is a backwards compatibility break and a rather subtle one at that. As such, such a change could probably only be made behind an ini setting, which would probably not help with your particular concern. ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=74312 -- Edit this bug report at https://bugs.php.net/bug.php?id=74312&edit=1