Bug #74005 [Opn->Csd]: mail.add_x_header causes RFC-breaking lone line feed, loss of subsequent header

Edit report at https://bugs.php.net/bug.php?id=74005&edit=1 ID: 74005 Updated by: ab@php.net Reported by: andy_schmidt at HM-Software dot com Summary: mail.add_x_header causes RFC-breaking lone line feed, loss of subsequent header -Status: Open +Status: Closed Type: Bug Package: Mail related Operating System: Windows PHP Version: 7.0.15 Block user comment: N Private report: N New Comment: Automatic comment on behalf of ab Revision: http://git.php.net/?p=php-src.git;a=commit;h=ec43a11581f457bd252d98e948d7a0531b4fdfc2 Log: Fixed bug #74005 mail.add_x_header causes RFC-breaking lone line feed Previous Comments: ------------------------------------------------------------------------ [2017-02-01 09:15:59] yohgaki@php.net Thank you for spotting what's wrong. It's easy bug to be fixed. ext/standard/mail.c if (PG(mail_x_header)) { const char *tmp = zend_get_executed_filename(); zend_string *f; f = php_basename(tmp, strlen(tmp), NULL, 0); if (headers != NULL && *headers) { spprintf(&hdr, 0, "X-PHP-Originating-Script: " ZEND_LONG_FMT ":%s\n%s", php_getuid(), ZSTR_VAL(f), headers); } else { spprintf(&hdr, 0, "X-PHP-Originating-Script: " ZEND_LONG_FMT ":%s", php_getuid(), ZSTR_VAL(f)); } zend_string_release(f); } All versions should use "\r\n" rather than "\n", including 5.6. BTW, I don't like line ending conversions. Suppose user script validate invalid "\r\n" in mail header string, e.g. mail address, but forgot to check "\r" and/or "\n", then line ending conversion by PHP would allow mail header injection. Recent mail() is made to reject multiple or broken line ending in extra headers, but it still allows "\n" for compatibility sake. This is irrelevant to this bug, just for the record. @anatol Since I'm not sure if this bug fix can be applied to 5.6, could you apply the fix? Mixing line ending chars is mess. It seems we are better to deprecate string extra headers and force users to use array extra headers someday. ------------------------------------------------------------------------ [2017-01-31 23:54:39] andy_schmidt at HM-Software dot com I think I have it worked out. The X-PHP... header likely had ALWAYS ended with a lone LF. However, with PHP 5 (and prior) either the mail function itself, or the win32 sendmail.c performed a "fixup" where it replaced all lone LFs with valid CRLFs to arrive at RFC compliant formatting. That made good sense, because Linux's own line-end is just LF, so the Linux mailers actually always expected single LFs, and had always outputted CRLF to the MTA. With PHP 7.0 that "fixup" is no longer performed with win32. The result is that your "own" malformed X-PHP... header, terminated by a lone LF, is now bleeding through to the MTA "unfixed". I have confirmed this behavior by testing THIS script under PHP 5 and PHP 7 - intentionally ending each additional header with a lone LF: $result = mail( "recipient@domain.com", "test subject", "test content", "X-PHP: ".phpversion()."\nFrom: sender@domain.com\n" ); Notice how I intentionally use malformed additional headers with just lone LFs! Result with PHP 5: Date: Tue, 31 Jan 2017 18:16:07 -0500\r\n Subject: test subject\r\n To: recipient@domain.com\r\n X-PHP-Originating-Script: 0:andytest.php\r\n X-PHP: 5.3.28\r\n From: sender@domain.com\r\n Result with PHP 7: Date: Tue, 31 Jan 2017 18:14:29 -0500\r\n Subject: test subject\r\n To: recipient@domain.com\r\n X-PHP-Originating-Script: 0:andytest.php\n X-PHP: 7.0.15\n From: sender@domain.com\r\n As this shows, with PHP 5 each individual occurrence of a lone LF in the headers is globally replaced with a proper CRLF (including your own X-PHP... header). With PHP 7, the various lone LF embedded in the additional headers are NO LONGER "fixed", only the FINAL LF (terminating the additional headers) is fixed to a CRLF. So this didn't break because a change to the X-PHP... header - it had always been "wrong". It broke because of a v7 change to mail() or win32 sendmail where it no longer corrects bad formatting. The greater impact of that is with WordPress and other PHP-based CMS systems under Windows. They will work fine under Linux, because there the lone LF are handled by the mailer. They will work fine under Windows with PHP 5, because it fixes the lone LFs to proper CRLF. However, if a Windows server is upgraded to PHP 7, THEN the various CMS will fail email delivery, because now the lone LFs are passed through to the MTA. For your information: WordPress (and others) use PHPmailer 5, and PHPmailer 5 uses mail(). ------------------------------------------------------------------------ [2017-01-31 18:45:00] andy_schmidt at HM-Software dot com Thanks for passing this on - THAT looks like a feasible test. Seeing that THAT didn't reproduce it made me take an extra step and it turns out the "duplicate From header" is actually a secondary problem, which is why you don't see it. I have identified the ACTUAL trigger by looking at the output in HEX. The real bug is with the mail.add_x_header config option in 7.0. Are you testing with 7.0? Then try outputting the headers (after DATA was sent) in HEX. In MY test 7.0 incorrectly ends the X-PHP header like THIS: X-PHP-Originating-Script: 0:andytest.php\n while 5.x correctly ends like THIS: X-PHP-Originating-Script: 0:andytest.php\r\n The problem is (once again) the infamous "lone line feed" which is NEVER permitted by SMTP. The secondary behavior is triggers is, that the default Windows MTA doesn't see a proper line-end at the end of the X-PHP header; the subsequent "From" header is therefor seen as being part of that same line. Because the output from PHP is lacking a recognizable, required "From" header, a default header is added by the MTA (not PHP, as previously assumed). Other (Linux based) mail systems DO treat a lone "LF" as a legitimate line end - and thus may choke on the duplicate From header they DO see. I suspect that you will be able to reproduce the original "lone linefeed" problem with 7.0, if your test outputs any \n and \r occurrences in HEX. (The duplicate "From" header is actually just a secondary problem.) I have confirmed that mail.add_x_header = Off will circumvent this bug under 7.0. ------------------------------------------------------------------------ [2017-01-31 17:45:42] ab@php.net Thanks for the further explanation, @andy_schmidt. Yes, i'm checking on Windows, and yes - i'm doing literally what you write to reproduce. I was asking you just where do you get the header list you post :) Now, I've put a primitive test facility, to see what is sent. Here you are https://github.com/php/php-src/commit/163bb87897c82eb7067e2a2e89818293a455e7a3 POST: 'HELO ' POST: 'MAIL FROM:<duplicated@test.com> ' POST: 'RCPT TO:<recipient@test.com> ' POST: 'DATA ' POST: 'Date: Tue, 31 Jan 2017 12:39:21 -0500 Subject: test subject To: recipient@test.com X-PHP-Originating-Script: 0:bug74005.php From: duplicated@test.com ' POST: ' ' POST: 'test content' POST: ' . ' POST: 'QUIT ' The PHP code is from your first post. Unfortunately, the way you write to be a reproducer doesn't work on my side. I see only one from header. Maybe it needs some real MTA or something else, that is not contained in your description. As long as the wrong behavior can't be reproduced and debugged, there's a little chance to understand, where the bug is, and to actually fix it. Please help me with that. Thanks. ------------------------------------------------------------------------ [2017-01-31 15:54:17] andy_schmidt at HM-Software dot com PS - rereading your note "PHP will send also MAIL FROM command, maybe that could cause some misbehavior with the MTA, as the header itself is sent, too. But also RCPT TO command is sent, so just to wonder why it wouldn't cause a duplicated to header", you seem to imply a relationship between headers and SMTP conversation that doesn't exist. Here a VERY common scenario. An email from "a" to "b" with a CC to "x". The headers for that email will say: From: a@sender.com To: b@recipient.com Cc: x@recipient.com With these headers (metadata), the two recipient will open the email in their clients and be presented the exact same from/to/cc information. However, the email is actually delivered through a mailing service or contact management system that uses bounces to clean up contact lists. When the email is delivered to "x", the following command sequence will be used: HELO MAIL FROM: bounce@sender.com RCPT TO: x@recipient.com DATA [followed by headers and email body] . QUIT As you can easily see, in this case the "From" and "To" header information does NOT appear in the SMTP commands AT ALL. The historic mail() function complicates matters because its parameter list was probably conceived before there was web hosting, e.g., at a time when "from" would be the system default, or could be automatically determined when the "sendmail" is launched in the user context. As a result, today we have a parameter list that is missing the crucial "From", and the mail function has to scramble that information from elsewhere. On the other hand, we have a "subject" parameter that clashes with a full set of other headers supplied as the next parameter. I'm not complaining - just trying to explain why things may seem a bit out of synch. ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=74005 -- Edit this bug report at https://bugs.php.net/bug.php?id=74005&edit=1