Bug #74005 [Csd]: mail.add_x_header causes RFC-breaking lone line feed, loss of subsequent header

Edit report at https://bugs.php.net/bug.php?id=74005&edit=1 ID: 74005 Updated by: ab@php.net Reported by: andy_schmidt at HM-Software dot com Summary: mail.add_x_header causes RFC-breaking lone line feed, loss of subsequent header Status: Closed Type: Bug Package: Mail related Operating System: Windows PHP Version: 7.0.15 -Assigned To: +Assigned To: ab Block user comment: N Private report: N New Comment: Yeah, that's the exact place, Yasuo. Clearly it's not RFC 5322 complaint, but not that easy. From the git history, see also bug #48620 and bug #50907 where it was already made to use CRLF and then reverted back to LF. Well, welcome to the real world :) We deal with a very legacy code here, so probably should be reluctant to to change this in the hurry. But also, i've found out, that there was a porting mistake, so then the headers in 7.0 was not noramlized anymore in win32/sendmail.c - pushed a fix touching that direction in first place. @andy_schmidt, thanks for the further investigation. Please check the snap from ec43a11581f457bd252d98e948d7a0531b4fdfc2 or any later. Also if anyone of the voters in this ticket could check as well, it'd be great. We miss the tests for this part badly, probably some could be written using a socket server as a dummy MTA. If someone has time to do taht right now, please file a PR! Thanks. Previous Comments: ------------------------------------------------------------------------ [2017-02-01 11:59:13] ab@php.net Automatic comment on behalf of ab Revision: http://git.php.net/?p=php-src.git;a=commit;h=ec43a11581f457bd252d98e948d7a0531b4fdfc2 Log: Fixed bug #74005 mail.add_x_header causes RFC-breaking lone line feed ------------------------------------------------------------------------ [2017-02-01 09:15:59] yohgaki@php.net Thank you for spotting what's wrong. It's easy bug to be fixed. ext/standard/mail.c if (PG(mail_x_header)) { const char *tmp = zend_get_executed_filename(); zend_string *f; f = php_basename(tmp, strlen(tmp), NULL, 0); if (headers != NULL && *headers) { spprintf(&hdr, 0, "X-PHP-Originating-Script: " ZEND_LONG_FMT ":%s\n%s", php_getuid(), ZSTR_VAL(f), headers); } else { spprintf(&hdr, 0, "X-PHP-Originating-Script: " ZEND_LONG_FMT ":%s", php_getuid(), ZSTR_VAL(f)); } zend_string_release(f); } All versions should use "\r\n" rather than "\n", including 5.6. BTW, I don't like line ending conversions. Suppose user script validate invalid "\r\n" in mail header string, e.g. mail address, but forgot to check "\r" and/or "\n", then line ending conversion by PHP would allow mail header injection. Recent mail() is made to reject multiple or broken line ending in extra headers, but it still allows "\n" for compatibility sake. This is irrelevant to this bug, just for the record. @anatol Since I'm not sure if this bug fix can be applied to 5.6, could you apply the fix? Mixing line ending chars is mess. It seems we are better to deprecate string extra headers and force users to use array extra headers someday. ------------------------------------------------------------------------ [2017-01-31 23:54:39] andy_schmidt at HM-Software dot com I think I have it worked out. The X-PHP... header likely had ALWAYS ended with a lone LF. However, with PHP 5 (and prior) either the mail function itself, or the win32 sendmail.c performed a "fixup" where it replaced all lone LFs with valid CRLFs to arrive at RFC compliant formatting. That made good sense, because Linux's own line-end is just LF, so the Linux mailers actually always expected single LFs, and had always outputted CRLF to the MTA. With PHP 7.0 that "fixup" is no longer performed with win32. The result is that your "own" malformed X-PHP... header, terminated by a lone LF, is now bleeding through to the MTA "unfixed". I have confirmed this behavior by testing THIS script under PHP 5 and PHP 7 - intentionally ending each additional header with a lone LF: $result = mail( "recipient@domain.com", "test subject", "test content", "X-PHP: ".phpversion()."\nFrom: sender@domain.com\n" ); Notice how I intentionally use malformed additional headers with just lone LFs! Result with PHP 5: Date: Tue, 31 Jan 2017 18:16:07 -0500\r\n Subject: test subject\r\n To: recipient@domain.com\r\n X-PHP-Originating-Script: 0:andytest.php\r\n X-PHP: 5.3.28\r\n From: sender@domain.com\r\n Result with PHP 7: Date: Tue, 31 Jan 2017 18:14:29 -0500\r\n Subject: test subject\r\n To: recipient@domain.com\r\n X-PHP-Originating-Script: 0:andytest.php\n X-PHP: 7.0.15\n From: sender@domain.com\r\n As this shows, with PHP 5 each individual occurrence of a lone LF in the headers is globally replaced with a proper CRLF (including your own X-PHP... header). With PHP 7, the various lone LF embedded in the additional headers are NO LONGER "fixed", only the FINAL LF (terminating the additional headers) is fixed to a CRLF. So this didn't break because a change to the X-PHP... header - it had always been "wrong". It broke because of a v7 change to mail() or win32 sendmail where it no longer corrects bad formatting. The greater impact of that is with WordPress and other PHP-based CMS systems under Windows. They will work fine under Linux, because there the lone LF are handled by the mailer. They will work fine under Windows with PHP 5, because it fixes the lone LFs to proper CRLF. However, if a Windows server is upgraded to PHP 7, THEN the various CMS will fail email delivery, because now the lone LFs are passed through to the MTA. For your information: WordPress (and others) use PHPmailer 5, and PHPmailer 5 uses mail(). ------------------------------------------------------------------------ [2017-01-31 18:45:00] andy_schmidt at HM-Software dot com Thanks for passing this on - THAT looks like a feasible test. Seeing that THAT didn't reproduce it made me take an extra step and it turns out the "duplicate From header" is actually a secondary problem, which is why you don't see it. I have identified the ACTUAL trigger by looking at the output in HEX. The real bug is with the mail.add_x_header config option in 7.0. Are you testing with 7.0? Then try outputting the headers (after DATA was sent) in HEX. In MY test 7.0 incorrectly ends the X-PHP header like THIS: X-PHP-Originating-Script: 0:andytest.php\n while 5.x correctly ends like THIS: X-PHP-Originating-Script: 0:andytest.php\r\n The problem is (once again) the infamous "lone line feed" which is NEVER permitted by SMTP. The secondary behavior is triggers is, that the default Windows MTA doesn't see a proper line-end at the end of the X-PHP header; the subsequent "From" header is therefor seen as being part of that same line. Because the output from PHP is lacking a recognizable, required "From" header, a default header is added by the MTA (not PHP, as previously assumed). Other (Linux based) mail systems DO treat a lone "LF" as a legitimate line end - and thus may choke on the duplicate From header they DO see. I suspect that you will be able to reproduce the original "lone linefeed" problem with 7.0, if your test outputs any \n and \r occurrences in HEX. (The duplicate "From" header is actually just a secondary problem.) I have confirmed that mail.add_x_header = Off will circumvent this bug under 7.0. ------------------------------------------------------------------------ [2017-01-31 17:45:42] ab@php.net Thanks for the further explanation, @andy_schmidt. Yes, i'm checking on Windows, and yes - i'm doing literally what you write to reproduce. I was asking you just where do you get the header list you post :) Now, I've put a primitive test facility, to see what is sent. Here you are https://github.com/php/php-src/commit/163bb87897c82eb7067e2a2e89818293a455e7a3 POST: 'HELO ' POST: 'MAIL FROM:<duplicated@test.com> ' POST: 'RCPT TO:<recipient@test.com> ' POST: 'DATA ' POST: 'Date: Tue, 31 Jan 2017 12:39:21 -0500 Subject: test subject To: recipient@test.com X-PHP-Originating-Script: 0:bug74005.php From: duplicated@test.com ' POST: ' ' POST: 'test content' POST: ' . ' POST: 'QUIT ' The PHP code is from your first post. Unfortunately, the way you write to be a reproducer doesn't work on my side. I see only one from header. Maybe it needs some real MTA or something else, that is not contained in your description. As long as the wrong behavior can't be reproduced and debugged, there's a little chance to understand, where the bug is, and to actually fix it. Please help me with that. Thanks. ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=74005 -- Edit this bug report at https://bugs.php.net/bug.php?id=74005&edit=1