#15630 [Com]: imap_utf7_decode appears to be broken

ID: 15630 Comment by: gamid@isayev.net Reported By: robert.marchand@umontreal.ca Status: Open Bug Type: IMAP related Operating System: SGI Irix 6.5 PHP Version: 4.2.2 New Comment: Robert Marchand wrote: > this will not work without changing current applications. Now it is not working at all for non-ASCII characters. Example: For IMAP folder name "test&WSc-" ("test" + chinese character) current imap_utf7_decode() returns "testY'" For IMAP folder name "testY'", current imap_utf7_decode() also returns "testY'" So, what you will do in this case? > The problem is that these function try to encode and decode without > knowing the charset used. 1) imap_utf7_decode() does not need to know charset of input string, because input string is encoded in modified UTF7 2) if you specify charset for imap_utf7_decode() output string, what will you do when IMAP folder name has characters from different charsets (example: "test&BCQA31kn-" - ASCII, Russian, German, Chinese)? > As it is now, 8 bit is expected from imap_utf7_decode. <...skiped...> > It should really be: > imap_utf7_utf8_decode > imap_utf7_utf16_decode (patched version) > imap_utf8_utf7_encode > imap_utf16_utf7_encode (patched version) I think you are confusing "8 bit" and UTF-8. UTF-8 encoded character is "8 bit" only for ASCII characters. For non-ASCII characters UTF-8 will be two and more bytes. So, imap_utf7_decode() != imap_utf7_utf8_decode(). Gamid Isayev Previous Comments: ------------------------------------------------------------------------ [2002-08-08 17:23:30] robert.marchand@umontreal.ca Hi, this will not work without changing current applications. As it is now, 8 bit is expected from imap_utf7_decode. The problem is that these function try to encode and decode without knowing the charset used. It should really be: imap_utf7_utf8_decode imap_utf7_utf16_decode (patched version) imap_utf8_utf7_encode imap_utf16_utf7_encode (patched version) Thanks. ------------------------------------------------------------------------ [2002-08-08 16:49:46] kalowsky@php.net Since I have no way to test this, can anyone else confirm or deny that this patch works? I'd rather not commit blindly. ------------------------------------------------------------------------ [2002-08-08 15:05:47] gamid@netilla.com Robert, The following patch fixes both imap_utf7_encode() and imap_utf7_decode() to work with UTF-16. PS: this patch is for PHP 4.2.2, the patch for CVS is posted in the php.dev Gamid Isayev --- php_imap.c Wed Aug 7 15:45:53 2002 +++ php_imap.c Thu Aug 8 14:24:16 2002 @@ -2215,14 +2215,14 @@ php_error(E_WARNING, "imap_utf7_decode: Invalid modified UTF-7 character: `%c'", *inp); RETURN_FALSE; } else if (*inp != '&') { - outlen++; + outlen += 2; } else if (inp + 1 == endp) { php_error(E_WARNING, "imap_utf7_decode: Unexpected end of string"); RETURN_FALSE; } else if (inp[1] != '-') { state = ST_DECODE0; } else { - outlen++; + outlen += 2; inp++; } } else if (*inp == '-') { @@ -2272,8 +2272,11 @@ if (*inp == '&' && inp[1] != '-') { state = ST_DECODE0; } - else if ((*outp++ = *inp) == '&') { - inp++; + else { + *outp++ = 0x00; + if ((*outp++ = *inp) == '&') { + inp++; + } } } else if (*inp == '-') { @@ -2349,29 +2352,42 @@ outlen = 0; state = ST_NORMAL; endp = (inp = in) + inlen; - while (inp < endp) { + while (inp < endp || state != ST_NORMAL) { if (state == ST_NORMAL) { - if (SPECIAL(*inp)) { + if (*inp == 0x00 && *(inp+1) < 0x80) { + /* ASCII character */ + outlen++; // for ASCII char + if (*(inp+1) == '&') + outlen++; // for '-' + inp += 2; + } else { + /* begin encoding */ state = ST_ENCODE0; - outlen++; - } else if (*inp++ == '&') { + outlen++; // for '&' + } + } else if (inp == endp || (*inp == 0x00 && *(inp+1) < 0x80)) { + /* flush overflow and terminate region */ + if (state != ST_ENCODE0) { outlen++; } - outlen++; - } else if (!SPECIAL(*inp)) { + outlen++; // for '-' state = ST_NORMAL; } else { - /* ST_ENCODE0 -> ST_ENCODE1 - two chars - * ST_ENCODE1 -> ST_ENCODE2 - one char - * ST_ENCODE2 -> ST_ENCODE0 - one char - */ - if (state == ST_ENCODE2) { - state = ST_ENCODE0; - } - else if (state++ == ST_ENCODE0) { - outlen++; + switch (state) { + case ST_ENCODE0: + outlen++; + state = ST_ENCODE1; + break; + case ST_ENCODE1: + outlen++; + state = ST_ENCODE2; + break; + case ST_ENCODE2: + outlen += 2; + state = ST_ENCODE0; + case ST_NORMAL: + break; } - outlen++; inp++; } } @@ -2388,14 +2404,17 @@ endp = (inp = in) + inlen; while (inp < endp || state != ST_NORMAL) { if (state == ST_NORMAL) { - if (SPECIAL(*inp)) { + if (*inp == 0x00 && *(inp+1) < 0x80) { + /* ASCII character */ + inp++; + if ((*outp++ = *inp++) == '&') + *outp++ = '-'; + } else { /* begin encoding */ *outp++ = '&'; state = ST_ENCODE0; - } else if ((*outp++ = *inp++) == '&') { - *outp++ = '-'; } - } else if (inp == endp || !SPECIAL(*inp)) { + } else if (inp == endp || (*inp == 0x00 && *(inp+1) < 0x80)) { /* flush overflow and terminate region */ if (state != ST_ENCODE0) { *outp++ = B64(*outp); ------------------------------------------------------------------------ [2002-08-07 11:30:22] gamid@isayev.net 1) Patched imap_utf7_decode() returns UTF-16 encoded string. So, to display string properly you should set charset to 'UTF-16' or convert UTF-16 string into preferred charset. 2) Current imap_utf7_encode() "converts 8bit data to modified UTF-7 text". Question is what mean "8bit data"? Right now I'm working on the patch for imap_utf7_encode() to add support for UTF-16 on input. It will allow do right covertion UTF-7 -> imap_utf7_decode() -> imap_utf7_encode() -> UTF-7. ------------------------------------------------------------------------ [2002-08-07 11:11:29] robert.marchand@umontreal.ca Hi, here is what I obtain with the patch applied (php-4.2.2): ---- output from Netscape 4.7: --------- folder (modified UTF-7): test&AN9ZJw- mb_convert_encoding test folder decoded: [testß?] encoded again: [test&AN9ZJw-] decoded again: [testß?] imap_utf7_decode test folder decoded: [ encoded again: [&AA-t&AA-e&AA-s&AA-t&ANE-Y$] decoded again: [ ----------------------------- Maybe there is something wrong with my installation here, but even then, I'm not sure adding zero bytes is the good solution. In my view it will break every application that now use these functions. At least this patch should be used only when --enable-mbstring is used. Thanks. ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at http://bugs.php.net/15630 -- Edit this bug report at http://bugs.php.net/?id=15630&edit=1