#15630 [Opn]: imap_utf7_decode appears to be broken

ID: 15630 Updated by: kalowsky@php.net Reported By: robert.marchand@umontreal.ca Status: Open Bug Type: IMAP related Operating System: SGI Irix 6.5 PHP Version: 4.2.2 New Comment: I've commited the SGI compiler patch to the CVS head. I have yet to see any real conclusion though on the utf7_decode() issue, and I really would not prefer to break BC. If there is some kind of agreement upon all whom this bug effects I'm open to it. Previous Comments: ------------------------------------------------------------------------ [2002-08-14 16:16:26] spc@sgi.com The patch added by Robert Marchand corrects a deficiency in the original code. The statement *outp++ |= outp[1] >> 2; in the ext/imap/php_imap.c file is clearly non-standard C. This means that the result is *compiler*-dependent, and no compiler can be considered "wrong". The C standard says that the order in which operands of an assignment operator are evaluated is undefined. In other words, it is equally correct for a compiler to produce code equivalent to temp = outp[1] >> 2; *outp++ |= temp; or to produce code equivalent to temp = outp; outp++; *temp |= outp[1] >> 2; Since these are not equivalent bits of code, the *source code* is wrong, not the compiler. Robert's patch suggested on 13 August corrects this non-standard source code. ------------------------------------------------------------------------ [2002-08-13 13:29:37] robert.marchand@umontreal.ca Hi, I've found the culprit in regards to the SGI problem. It is related to auto-increment operator and complex assignment. This doesn't work on SGI: *outp++ |= outp[1] >> 2; Here is my patch that correct the two function on SGI with the SGI Compiler (MIPSPRO): --- php_imap.c.nowarn Tue Jul 30 10:04:24 2002 +++ php_imap.c Tue Aug 13 11:44:50 2002 @@ -2187,6 +2187,7 @@ zval **arg; const unsigned char *in, *inp, *endp; unsigned char *out, *outp; + unsigned char c; int inlen, outlen; enum { ST_NORMAL, /* printable text */ @@ -2289,13 +2290,15 @@ break; case ST_DECODE1: outp[1] = UNB64(*inp); - *outp++ |= outp[1] >> 4; + c = outp[1] >> 4; + *outp++ |= c; *outp <<= 4; state = ST_DECODE2; break; case ST_DECODE2: outp[1] = UNB64(*inp); - *outp++ |= outp[1] >> 2; + c = outp[1] >> 2; + *outp++ |= c; *outp <<= 6; state = ST_DECODE3; break; @@ -2329,6 +2332,7 @@ zval **arg; const unsigned char *in, *inp, *endp; unsigned char *out, *outp; + unsigned char c; int inlen, outlen; enum { ST_NORMAL, /* printable text */ @@ -2399,7 +2403,8 @@ } else if (inp == endp || !SPECIAL(*inp)) { /* flush overflow and terminate region */ if (state != ST_ENCODE0) { - *outp++ = B64(*outp); + c = B64(*outp); + *outp++ = c; } *outp++ = '-'; state = ST_NORMAL; @@ -2412,12 +2417,14 @@ state = ST_ENCODE1; break; case ST_ENCODE1: - *outp++ = B64(*outp | *inp >> 4); + c = B64(*outp | *inp >> 4); + *outp++ = c; *outp = *inp++ << 2; state = ST_ENCODE2; break; case ST_ENCODE2: - *outp++ = B64(*outp | *inp >> 6); + c = B64(*outp | *inp >> 6); + *outp++ = c; *outp++ = B64(*inp++); state = ST_ENCODE0; case ST_NORMAL: This patch was applied to the original php_imap.c (4.2.2) but it can also be applied to the new version from Gamid Isayev. Thanks. ------------------------------------------------------------------------ [2002-08-12 15:47:27] robert.marchand@umontreal.ca Hi, this is to confirm that there is a SGI specific issue because I tested the new patch on a Linux Redhat 7.2 System with PHP 4.2.2 compiled manually. Here is the output from the test: folder (modified UTF-7): &AMk-l&AOk-ments envoy&AOk-s mb_convert_encoding test folder decoded: [Ã?léments envoyés] (hexa: c3 89 6c c3 a9 6d 65 6e 74 73 20 65 6e 76 6f 79 c3 a9 73 ) encoded again: [&AMk-l&AOk-ments envoy&AOk-s] decoded again: [Ã?léments envoyés] (hexa: c3 89 6c c3 a9 6d 65 6e 74 73 20 65 6e 76 6f 79 c3 a9 73 ) imap_utf7_decode test folder decoded: [ (hexa: 0 c9 0 6c 0 e9 0 6d 0 65 0 6e 0 74 0 73 0 20 0 65 0 6e 0 76 0 6f 0 79 0 e9 0 73 ) encoded again: [&AMk-l&AOk-ments envoy&AOk-s] decoded again: [ (hexa: 0 c9 0 6c 0 e9 0 6d 0 65 0 6e 0 74 0 73 0 20 0 65 0 6e 0 76 0 6f 0 79 0 e9 0 73 ) I have also try on an O2 SGI box (it is 32 bit) and it is the same as all SGI boxes I have tried. I'll take a look when I have a moment. Thanks. ------------------------------------------------------------------------ [2002-08-12 13:57:01] robert.marchand@umontreal.ca Hi, you're write about the "utf8" thing. I was meaning "8bit". For the rest, I cannot change the software I use (Horde/IMP) because it is not me who wrote it. I can assure you it will break if you go with your mods. This is for the general problem. Now it seems I have a specific problem here with my SGI platform. Here is what I get with your patch: folder (modified UTF-7): test&AN9ZJw- mb_convert_encoding test folder decoded: [testÃY大] (hexa: 74 65 73 74 c3 9f e5 a4 a7 ) encoded again: [test&AN9ZJw-] decoded again: [testÃY大] (hexa: 74 65 73 74 c3 9f e5 a4 a7 ) imap_utf7_decode test folder decoded: [ (hexa: 0 74 0 65 0 73 0 74 0 d0 59 24 ) encoded again: [test&ANBZJA-] decoded again: [ (hexa: 0 74 0 65 0 73 0 74 5 d9 59 24 ) Here is another sample: folder (modified UTF-7): &AMk-l&AOk-ments envoy&AOk-s mb_convert_encoding test folder decoded: [Ã?léments envoyés] (hexa: c3 89 6c c3 a9 6d 65 6e 74 73 20 65 6e 76 6f 79 c3 a9 73 ) encoded again: [&AMk-l&AOk-ments envoy&AOk-s] decoded again: [Ã?léments envoyés] (hexa: c3 89 6c c3 a9 6d 65 6e 74 73 20 65 6e 76 6f 79 c3 a9 73 ) imap_utf7_decode test folder decoded: [ Ó (hexa: 6 d3 0 6c 0 e0 0 6d 0 65 0 6e 0 74 0 73 0 20 0 65 0 6e 0 76 0 6f 0 79 0 e0 0 73 ) encoded again: [&BPp-l&A,g-ments envoy&Afa-s] decoded again: [ û (hexa: 4 fb 0 6c 0 fb 0 6d 0 65 0 6e 0 74 0 73 0 20 0 65 0 6e 0 76 0 6f 0 79 0 fc 0 73 ) Here is the PHP test page to generate this output: <HTML> <HEAD> <TITLE>Test UTF7</TITLE> <META HTTP-EQUIV="Content-Type" CONTENT="text/html;charset=utf-16"> </HEAD> <BODY> <? function hexstr($s) { echo "(hexa: "; for ($i=0;$i<strlen($s);$i++) { echo dechex(ord($s[$i])), " "; } echo ")<br>"; } //$folder = 'test&AN9ZJw-'; $folder = '&AMk-l&AOk-ments envoy&AOk-s'; echo "folder (modified UTF-7): $folder<BR><BR>\n"; echo "<strong>mb_convert_encoding test</strong><BR>\n"; $test = $folder; $test = mb_convert_encoding($test, "UTF-8", "UTF7-IMAP"); echo " folder decoded: [$test]<BR>\n"; hexstr($test); $test = mb_convert_encoding($test, "UTF7-IMAP", "UTF-8"); echo "encoded again: [", $test, "]<BR>\n"; $test = mb_convert_encoding($test, "UTF-8", "UTF7-IMAP"); echo "decoded again: [", $test, "]<BR>\n"; hexstr($test); echo "<BR><strong>imap_utf7_decode test</strong><BR>\n"; $test = $folder; $test = imap_utf7_decode($test); echo "folder decoded: [", $test, "]<BR>\n"; hexstr($test); $test = imap_utf7_encode($test); echo "encoded again: [", $test, "]<BR>\n"; $test = imap_utf7_decode($test); echo "decoded again: [", $test, "]<BR>\n"; hexstr($test); ?> </BODY> </HTML> I am on a 64 bit platform. Could this be related to a wrapping shift? There is definitly something wrong here. Thanks. ------------------------------------------------------------------------ [2002-08-09 10:57:10] gamid@isayev.net Robert Marchand wrote: > this will not work without changing current applications. Now it is not working at all for non-ASCII characters. Example: For IMAP folder name "test&WSc-" ("test" + chinese character) current imap_utf7_decode() returns "testY'" For IMAP folder name "testY'", current imap_utf7_decode() also returns "testY'" So, what you will do in this case? > The problem is that these function try to encode and decode without > knowing the charset used. 1) imap_utf7_decode() does not need to know charset of input string, because input string is encoded in modified UTF7 2) if you specify charset for imap_utf7_decode() output string, what will you do when IMAP folder name has characters from different charsets (example: "test&BCQA31kn-" - ASCII, Russian, German, Chinese)? > As it is now, 8 bit is expected from imap_utf7_decode. <...skiped...> > It should really be: > imap_utf7_utf8_decode > imap_utf7_utf16_decode (patched version) > imap_utf8_utf7_encode > imap_utf16_utf7_encode (patched version) I think you are confusing "8 bit" and UTF-8. UTF-8 encoded character is "8 bit" only for ASCII characters. For non-ASCII characters UTF-8 will be two and more bytes. So, imap_utf7_decode() != imap_utf7_utf8_decode(). Gamid Isayev ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at http://bugs.php.net/15630 -- Edit this bug report at http://bugs.php.net/?id=15630&edit=1