Bug #75131 [NEW]: Iconv output buffering handler doesn't encode correctly across chunk boundaries

From: jocrutrisi at ibsats dot com Operating system: Linux PHP version: 7.2.0beta3 Package: ICONV related Bug Type: Bug Bug description:Iconv output buffering handler doesn't encode correctly across chunk boundaries Description: ------------ The "ob_iconv_handler" is the only means in PHP to convert streaming input from one charset to another charset. We can't use any of the other functions, because as we process in chunks of undefined or fixed size, variable-width encodings like UTF-8 often end up with partial characters at the boundaries (start/end) of the chunk. So if we convert every chunk as if it's a complete string, we'll end up with corrupted output as the conversion can't see the entire characters. Normally since "ob_iconv_handler" is advertized as a stream output handler, you'd think it handles this scenario correctly, alas it doesn't. Output is correctly encoded when we give "complete" chunks to the handler. But if we cut down the buffer size so partial characters are sent in each chunk... we get garbled output. This is especially troubling not only because it's not correct, but because this is supposedly the ONLY WAY to convert a stream from one charset to another. The only option in PHP right now, is to put an entire string in memory and convert it this way. If it doesn't fit in memory we're S.O.L. There are many other issues with ob_iconv_handler - it relies on global settings, it has global state and is not reentrant... It'd be amazing if we had some sort of iconv_open iconv_read/write iconv_close API to handle these cases, but I digress... Find the examples below reproducing the problem. Test script: --------------- -------------------------------------------------------------------------------- EXAMPLE 2: -------------------------------------------------------------------------------- // Make sure display is right for browsers (also works in CLI if UTF8 is supported). header('Content-Type: text/plain; charset=utf-8'); // UTF-8 sample text. $t = 'Здравей!'; // We convert it to UTF-16LE, to then convert it back to UTF-8 $t = iconv('UTF-8', 'UTF-16LE', $t); // We set-up ob_iconv_handler() to do UTF-16LE -> UTF-8 conversion ini_set('internal_encoding', 'UTF-16LE'); ini_set('output_encoding', 'UTF-8'); // Prints "Здравей!" as expected. ob_start('ob_iconv_handler', 4096); for ($i = 0; $i<strlen($t); $i++) { echo $t[$i]; } ob_end_flush(); -------------------------------------------------------------------------------- EXAMPLE 2: -------------------------------------------------------------------------------- // Make sure display is right for browsers (also works in CLI if UTF8 is supported). header('Content-Type: text/plain; charset=utf-8'); // UTF-8 sample text. $t = 'Здравей!'; // We convert it to UTF-16LE, to then convert it back to UTF-8 $t = iconv('UTF-8', 'UTF-16LE', $t); // We set-up ob_iconv_handler() to do UTF-16LE -> UTF-8 conversion ini_set('internal_encoding', 'UTF-16LE'); ini_set('output_encoding', 'UTF-8'); // Prints GARBLED OUTPUT. ob_start('ob_iconv_handler', 1); for ($i = 0; $i<strlen($t); $i++) { echo $t[$i]; } ob_end_flush(); Expected result: ---------------- Same correct output in both samples. Actual result: -------------- Garbled output in the second example. -- Edit bug report at https://bugs.php.net/bug.php?id=75131&edit=1 -- Try a snapshot (PHP 5.4): https://bugs.php.net/fix.php?id=75131&r=trysnapshot54 Try a snapshot (PHP 5.5): https://bugs.php.net/fix.php?id=75131&r=trysnapshot55 Try a snapshot (trunk): https://bugs.php.net/fix.php?id=75131&r=trysnapshottrunk Fixed in SVN: https://bugs.php.net/fix.php?id=75131&r=fixed Fixed in release: https://bugs.php.net/fix.php?id=75131&r=alreadyfixed Need backtrace: https://bugs.php.net/fix.php?id=75131&r=needtrace Need Reproduce Script: https://bugs.php.net/fix.php?id=75131&r=needscript Try newer version: https://bugs.php.net/fix.php?id=75131&r=oldversion Not developer issue: https://bugs.php.net/fix.php?id=75131&r=support Expected behavior: https://bugs.php.net/fix.php?id=75131&r=notwrong Not enough info: https://bugs.php.net/fix.php?id=75131&r=notenoughinfo Submitted twice: https://bugs.php.net/fix.php?id=75131&r=submittedtwice register_globals: https://bugs.php.net/fix.php?id=75131&r=globals PHP 4 support discontinued: https://bugs.php.net/fix.php?id=75131&r=php4 Daylight Savings: https://bugs.php.net/fix.php?id=75131&r=dst IIS Stability: https://bugs.php.net/fix.php?id=75131&r=isapi Install GNU Sed: https://bugs.php.net/fix.php?id=75131&r=gnused Floating point limitations: https://bugs.php.net/fix.php?id=75131&r=float No Zend Extensions: https://bugs.php.net/fix.php?id=75131&r=nozend MySQL Configuration Error: https://bugs.php.net/fix.php?id=75131&r=mysqlcfg