Bug #74005 [Fbk->Opn]: mail.add_x_header causes RFC-breaking lone line feed, loss of subsequent header

Edit report at https://bugs.php.net/bug.php?id=74005&edit=1 ID: 74005 User updated by: andy_schmidt at HM-Software dot com Reported by: andy_schmidt at HM-Software dot com -Summary: mail function duplicates FROM header +Summary: mail.add_x_header causes RFC-breaking lone line feed, loss of subsequent header -Status: Feedback +Status: Open Type: Bug Package: Mail related Operating System: Windows PHP Version: 7.0.15 Block user comment: N Private report: N New Comment: Thanks for passing this on - THAT looks like a feasible test. Seeing that THAT didn't reproduce it made me take an extra step and it turns out the "duplicate From header" is actually a secondary problem, which is why you don't see it. I have identified the ACTUAL trigger by looking at the output in HEX. The real bug is with the mail.add_x_header config option in 7.0. Are you testing with 7.0? Then try outputting the headers (after DATA was sent) in HEX. In MY test 7.0 incorrectly ends the X-PHP header like THIS: X-PHP-Originating-Script: 0:andytest.php\n while 5.x correctly ends like THIS: X-PHP-Originating-Script: 0:andytest.php\r\n The problem is (once again) the infamous "lone line feed" which is NEVER permitted by SMTP. The secondary behavior is triggers is, that the default Windows MTA doesn't see a proper line-end at the end of the X-PHP header; the subsequent "From" header is therefor seen as being part of that same line. Because the output from PHP is lacking a recognizable, required "From" header, a default header is added by the MTA (not PHP, as previously assumed). Other (Linux based) mail systems DO treat a lone "LF" as a legitimate line end - and thus may choke on the duplicate From header they DO see. I suspect that you will be able to reproduce the original "lone linefeed" problem with 7.0, if your test outputs any \n and \r occurrences in HEX. (The duplicate "From" header is actually just a secondary problem.) I have confirmed that mail.add_x_header = Off will circumvent this bug under 7.0. Previous Comments: ------------------------------------------------------------------------ [2017-01-31 17:45:42] ab@php.net Thanks for the further explanation, @andy_schmidt. Yes, i'm checking on Windows, and yes - i'm doing literally what you write to reproduce. I was asking you just where do you get the header list you post :) Now, I've put a primitive test facility, to see what is sent. Here you are https://github.com/php/php-src/commit/163bb87897c82eb7067e2a2e89818293a455e7a3 POST: 'HELO ' POST: 'MAIL FROM:<duplicated@test.com> ' POST: 'RCPT TO:<recipient@test.com> ' POST: 'DATA ' POST: 'Date: Tue, 31 Jan 2017 12:39:21 -0500 Subject: test subject To: recipient@test.com X-PHP-Originating-Script: 0:bug74005.php From: duplicated@test.com ' POST: ' ' POST: 'test content' POST: ' . ' POST: 'QUIT ' The PHP code is from your first post. Unfortunately, the way you write to be a reproducer doesn't work on my side. I see only one from header. Maybe it needs some real MTA or something else, that is not contained in your description. As long as the wrong behavior can't be reproduced and debugged, there's a little chance to understand, where the bug is, and to actually fix it. Please help me with that. Thanks. ------------------------------------------------------------------------ [2017-01-31 15:54:17] andy_schmidt at HM-Software dot com PS - rereading your note "PHP will send also MAIL FROM command, maybe that could cause some misbehavior with the MTA, as the header itself is sent, too. But also RCPT TO command is sent, so just to wonder why it wouldn't cause a duplicated to header", you seem to imply a relationship between headers and SMTP conversation that doesn't exist. Here a VERY common scenario. An email from "a" to "b" with a CC to "x". The headers for that email will say: From: a@sender.com To: b@recipient.com Cc: x@recipient.com With these headers (metadata), the two recipient will open the email in their clients and be presented the exact same from/to/cc information. However, the email is actually delivered through a mailing service or contact management system that uses bounces to clean up contact lists. When the email is delivered to "x", the following command sequence will be used: HELO MAIL FROM: bounce@sender.com RCPT TO: x@recipient.com DATA [followed by headers and email body] . QUIT As you can easily see, in this case the "From" and "To" header information does NOT appear in the SMTP commands AT ALL. The historic mail() function complicates matters because its parameter list was probably conceived before there was web hosting, e.g., at a time when "from" would be the system default, or could be automatically determined when the "sendmail" is launched in the user context. As a result, today we have a parameter list that is missing the crucial "From", and the mail function has to scramble that information from elsewhere. On the other hand, we have a "subject" parameter that clashes with a full set of other headers supplied as the next parameter. I'm not complaining - just trying to explain why things may seem a bit out of synch. ------------------------------------------------------------------------ [2017-01-31 15:14:22] andy_schmidt at HM-Software dot com Hi - can you please confirm that you ARE testing this on a Windows system? The implementation of mail() in Windows differs from the Linux implementation in SEVERAL aspects! For one, Windows uses direct SMTP delivery instead of a a "mailer" (such as "sendmail"). I ran another test that might help narrow down at what point the extra From: header is generated. $result = mail( "parmto@domain.com", "test subject", "test content", "From: PFHeader <hdrfrom@domain.com>\r\nTo: PTHeader <hdrto@domain.com>\r\n\Subject: hdrsubject\r\n" ); produces this: Date: Tue, 31 Jan 2017 09:15:35 -0500 Subject: test subject X-PHP-Originating-Script: 0:andytest.php From: PFHeader <hdrfrom@domain.com> To: PTHeader <hdrto@domain.com> Subject: hdrsubject From: hdrfrom@domain.com It shows that the Date, Subject and X-PHP headers come first, so they seem to be created during the time when the parameter list and the config settings are processed. You can also see that the next step is that the supplied headers are "appended as is", including any (unnecessary) "Subject" header and the bracketed "From" and "To" headers that each have the "name" component. THREE things are important to note from this: 1. My original minimal test had supplied no "To:" header, so one had been created between the "Subject" and "X-PHP..." header. It's clear from this second test, that the generating of the "To:" header is done conditionally, after checking if such a header already exists. (The same sanity check is NOT done for the Subject header, FWIW) 2. The troublesome second "From" header is generated AFTER the headers from the parameters had been appended. If I look at this from a programming perspective, I would theorize that there are four distinct phases in the code: a) Parameter are processed. The "to" information is stored away so it can be used later for the RCPT TO commands. b) Date and Subject headers are generated unconditionally, a "To" header is generated if none is in the "headers" parameter, the X-PHP header is generated based on config settings. c) The "headers" parameter is appended AS IS. d) The SMTP commands are prepared. The comma-separated list of "to" parameters are turned into RCPT TO commands. Either already during the first phase or only now the "headers" parameter list is parsed for "From" information. Only the unbracketed email address portion is extracted. The MAIL FROM command is generated - and a matching (duplicate) "From:" header is appended to the headers. Since the preparation/sending of SMTP commands via TCP/IP is unique to the Windows implementation, you will not reproduce that elsewhere! This happens to be my area of expertise. Allow me to add some detail to avoid any miscommunication. Feel free to skip over this if the SMTP protocol and RFCs are your home-turf as well. "HELO/EHLO", "MAIL FROM", "RCPT TO", "DATA" are not "generated" in the same sense that the headers are "generated", but they are the actual commands being issued via TCP/IP. These commands ONLY affect how the email is routed to the final recipient. The headers do NOT. On the other hand, these commands do not travel with/as part of the email. Instead each "hop" along the way, will issue its own/new set/subset of commands as it relays the message data to the next agent. (Per example 5 "RCPT TO" commands originally sent by PHP, will eventually result 5 copies of the DATA being sent with 1 RCPT TO command each to 5 different servers handling 5 different recipient domains!) On the other hand, the "To:", "From:", "CC:" headers are completely embedded inside the data - and are entirely irrelevant to the sending/relaying process and often aren't even factual (e.g., don't match the MAIL FROM, RCTP TO literally). They are just "metadata" that the recipient's email client will use to display the "apparent" sender/recipient, subject, data, etc. So, it's not only normal but absolutely necessary, that you will have MAIL FROM and RCPT TO commands taking care of delivery, and ALSO to have To/From headers supplying metadata to the email client software of the recipient. Only AFTER the emails have reached the final destination, will a server (like Gmails) peek inside the email data to see if the content looks legitimate. Besides scanning the email body and attachments, the headers are helpful in this process - because imperfect/deceptive/mismatching headers hint at a possibly illegitimate message. Which is what's happening here. I have observed the headers generated by PHP by looking at the DATA message stream that the mail server spools to disk, so that it can be relayed to the next/destination server. When the test script is executed by PHP 5, it outputs DATA that has only the ONE "From:" from the mail() parameter list, if the test script is executed by PHP 7, it has a second "From:". I did notice in the change log that the mail function (specially handling of headers) has seen some significant work - so I'm not surprised that something could have been introduced accidentally. ------------------------------------------------------------------------ [2017-01-31 00:43:10] ab@php.net I was asking for only the relevant ini, that would pass into a post, but thanks for sharing anyway. Unfortunately i still don't reproduce the behavior,the only diff option in your ini is mail.add_x_header, as far as i see. In PHP 5 the code flow is same. Still PHP will send also MAIL FROM command, maybe that could cause some misbehavior with the MTA, as the header itself is sent, too. But also RCPT TO command is sent, so just to wonder why it wouldn't cause a duplicated to header. I still have to ask for more info on how to reproduce this behavior. Where do you get the header list from? Thanks. ------------------------------------------------------------------------ [2017-01-30 14:10:52] andy_schmidt at HM-Software dot com I tried to post the PHP.INI content, but your web site responds with: "ERROR: Your comment looks like SPAM by its content. Please consider rewording." and I see no "upload/file attach" button on your form. On to plan "C"... The following download link will be valid for ONE week: https://mediacenter.email/download/2072a12607534fca90f2a065223/9f5bcbd3 ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=74005 -- Edit this bug report at https://bugs.php.net/bug.php?id=74005&edit=1