Bug #74005 [Csd]: mail.add_x_header causes RFC-breaking lone line feed, loss of subsequent header

Edit report at https://bugs.php.net/bug.php?id=74005&edit=1 ID: 74005 User updated by: andy_schmidt at HM-Software dot com Reported by: andy_schmidt at HM-Software dot com Summary: mail.add_x_header causes RFC-breaking lone line feed, loss of subsequent header Status: Closed Type: Bug Package: Mail related Operating System: Windows PHP Version: 7.0.15 Assigned To: ab Block user comment: N Private report: N New Comment: I probably wouldn't backport this to PHP 5, because in PHP 5 you were still fixing lone line feeds in the win32 implementation. So the bad X-PHP... header had no ill effect. Under PHP 7 there is need to action because now the headers are "passed through" under win32 - so they HAVE to be SMTP compliant. I have reviewed the earlier patch you cited: https://bugs.php.net/bug.php?id=48620 It shows the original code to have been: spprintf(&hdr, 0, "%s\r\nX-PHP-Originating-Script: %ld:%s\n", headers, php_getuid(), f); so it unconditionally bracketed the X-PHP header with a leading CRLF but trailing lone LF!? This was REALLY bad code - because it was following NEITHER convention. With an old Linux mailer, the leading CRLF would be treated as two line feeds. It also caused the other reported problem if there were no user supplied headers in which case the (sole) X-PHP header would start with the CRLF (essentially creating an empty header)! The attempted fix only sought to address the second scenario by NOT starting with a CRLF if a previous header existed. It still kept the inconsistency of a leading CRLF but a trailing lone LF. Your "legacy" challenge is reconciling two directly opposing requirements: - mail() under Linux was talking to a mailer, some of which had required the input to be operating system line end (single line feed) and rejected/mangled any SMTP compliant CR/LF. - mail() under win32 on the other hand is talking directly to SMTP, which absolutely mandates RFC compliant line-ends. Before PHP 7 you had taken the logical/pragmatic step of simply adapting the win32 implementation to fix any lone LF (or CR or LFCR?) to CRLF (which is a simple RegEx and kept everyone happy). If you no longer want to do that for PHP 7, then you would need to reject non-RFC compliant additionalheaders (including your own broken X-PHP... header!). But, I suspect you'd effect a lot of existing applications that relied on single line feeds. Alternatively you would have to reinstate the PHP 5 behavior for win 32 thus once again aligning the behavior for both operating system families until you are ready to complete change the additionalheaders from "string" to "array". Previous Comments: ------------------------------------------------------------------------ [2017-02-01 12:08:30] ab@php.net Yeah, that's the exact place, Yasuo. Clearly it's not RFC 5322 complaint, but not that easy. From the git history, see also bug #48620 and bug #50907 where it was already made to use CRLF and then reverted back to LF. Well, welcome to the real world :) We deal with a very legacy code here, so probably should be reluctant to to change this in the hurry. But also, i've found out, that there was a porting mistake, so then the headers in 7.0 was not noramlized anymore in win32/sendmail.c - pushed a fix touching that direction in first place. @andy_schmidt, thanks for the further investigation. Please check the snap from ec43a11581f457bd252d98e948d7a0531b4fdfc2 or any later. Also if anyone of the voters in this ticket could check as well, it'd be great. We miss the tests for this part badly, probably some could be written using a socket server as a dummy MTA. If someone has time to do taht right now, please file a PR! Thanks. ------------------------------------------------------------------------ [2017-02-01 11:59:13] ab@php.net Automatic comment on behalf of ab Revision: http://git.php.net/?p=php-src.git;a=commit;h=ec43a11581f457bd252d98e948d7a0531b4fdfc2 Log: Fixed bug #74005 mail.add_x_header causes RFC-breaking lone line feed ------------------------------------------------------------------------ [2017-02-01 09:15:59] yohgaki@php.net Thank you for spotting what's wrong. It's easy bug to be fixed. ext/standard/mail.c if (PG(mail_x_header)) { const char *tmp = zend_get_executed_filename(); zend_string *f; f = php_basename(tmp, strlen(tmp), NULL, 0); if (headers != NULL && *headers) { spprintf(&hdr, 0, "X-PHP-Originating-Script: " ZEND_LONG_FMT ":%s\n%s", php_getuid(), ZSTR_VAL(f), headers); } else { spprintf(&hdr, 0, "X-PHP-Originating-Script: " ZEND_LONG_FMT ":%s", php_getuid(), ZSTR_VAL(f)); } zend_string_release(f); } All versions should use "\r\n" rather than "\n", including 5.6. BTW, I don't like line ending conversions. Suppose user script validate invalid "\r\n" in mail header string, e.g. mail address, but forgot to check "\r" and/or "\n", then line ending conversion by PHP would allow mail header injection. Recent mail() is made to reject multiple or broken line ending in extra headers, but it still allows "\n" for compatibility sake. This is irrelevant to this bug, just for the record. @anatol Since I'm not sure if this bug fix can be applied to 5.6, could you apply the fix? Mixing line ending chars is mess. It seems we are better to deprecate string extra headers and force users to use array extra headers someday. ------------------------------------------------------------------------ [2017-01-31 23:54:39] andy_schmidt at HM-Software dot com I think I have it worked out. The X-PHP... header likely had ALWAYS ended with a lone LF. However, with PHP 5 (and prior) either the mail function itself, or the win32 sendmail.c performed a "fixup" where it replaced all lone LFs with valid CRLFs to arrive at RFC compliant formatting. That made good sense, because Linux's own line-end is just LF, so the Linux mailers actually always expected single LFs, and had always outputted CRLF to the MTA. With PHP 7.0 that "fixup" is no longer performed with win32. The result is that your "own" malformed X-PHP... header, terminated by a lone LF, is now bleeding through to the MTA "unfixed". I have confirmed this behavior by testing THIS script under PHP 5 and PHP 7 - intentionally ending each additional header with a lone LF: $result = mail( "recipient@domain.com", "test subject", "test content", "X-PHP: ".phpversion()."\nFrom: sender@domain.com\n" ); Notice how I intentionally use malformed additional headers with just lone LFs! Result with PHP 5: Date: Tue, 31 Jan 2017 18:16:07 -0500\r\n Subject: test subject\r\n To: recipient@domain.com\r\n X-PHP-Originating-Script: 0:andytest.php\r\n X-PHP: 5.3.28\r\n From: sender@domain.com\r\n Result with PHP 7: Date: Tue, 31 Jan 2017 18:14:29 -0500\r\n Subject: test subject\r\n To: recipient@domain.com\r\n X-PHP-Originating-Script: 0:andytest.php\n X-PHP: 7.0.15\n From: sender@domain.com\r\n As this shows, with PHP 5 each individual occurrence of a lone LF in the headers is globally replaced with a proper CRLF (including your own X-PHP... header). With PHP 7, the various lone LF embedded in the additional headers are NO LONGER "fixed", only the FINAL LF (terminating the additional headers) is fixed to a CRLF. So this didn't break because a change to the X-PHP... header - it had always been "wrong". It broke because of a v7 change to mail() or win32 sendmail where it no longer corrects bad formatting. The greater impact of that is with WordPress and other PHP-based CMS systems under Windows. They will work fine under Linux, because there the lone LF are handled by the mailer. They will work fine under Windows with PHP 5, because it fixes the lone LFs to proper CRLF. However, if a Windows server is upgraded to PHP 7, THEN the various CMS will fail email delivery, because now the lone LFs are passed through to the MTA. For your information: WordPress (and others) use PHPmailer 5, and PHPmailer 5 uses mail(). ------------------------------------------------------------------------ [2017-01-31 18:45:00] andy_schmidt at HM-Software dot com Thanks for passing this on - THAT looks like a feasible test. Seeing that THAT didn't reproduce it made me take an extra step and it turns out the "duplicate From header" is actually a secondary problem, which is why you don't see it. I have identified the ACTUAL trigger by looking at the output in HEX. The real bug is with the mail.add_x_header config option in 7.0. Are you testing with 7.0? Then try outputting the headers (after DATA was sent) in HEX. In MY test 7.0 incorrectly ends the X-PHP header like THIS: X-PHP-Originating-Script: 0:andytest.php\n while 5.x correctly ends like THIS: X-PHP-Originating-Script: 0:andytest.php\r\n The problem is (once again) the infamous "lone line feed" which is NEVER permitted by SMTP. The secondary behavior is triggers is, that the default Windows MTA doesn't see a proper line-end at the end of the X-PHP header; the subsequent "From" header is therefor seen as being part of that same line. Because the output from PHP is lacking a recognizable, required "From" header, a default header is added by the MTA (not PHP, as previously assumed). Other (Linux based) mail systems DO treat a lone "LF" as a legitimate line end - and thus may choke on the duplicate From header they DO see. I suspect that you will be able to reproduce the original "lone linefeed" problem with 7.0, if your test outputs any \n and \r occurrences in HEX. (The duplicate "From" header is actually just a secondary problem.) I have confirmed that mail.add_x_header = Off will circumvent this bug under 7.0. ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=74005 -- Edit this bug report at https://bugs.php.net/bug.php?id=74005&edit=1