Req #80689 [NEW]: Add support for incremental encoding conversion

From: dhammond at webdevout dot net Operating system: PHP version: 8.0.1 Package: mbstring related Bug Type: Feature/Change Request Bug description:Add support for incremental encoding conversion Description: ------------ Mbstring currently supports converting a complete string of text from one encoding to another, but it doesn't yet support converting a stream of text incrementally. This is needed in streaming workflows that process one chunk of bytes at a time. If you try to use mb_convert_encoding() in a streaming workflow, you run into problems with multibyte encodings: 1. The chunk might end in the middle of a multibyte sequence, resulting in corruption at the chunk boundaries. 2. Byte order detection in encodings like UTF-16 gets reset each chunk, meaning it might correctly interpret the first chunk as UTF-16LE and then incorrectly interpret the next chunk as UTF-16BE. 3. Some special encodings, like BASE64, have unique problems at chunk boundaries. In the case of BASE64 output encoding, if the input chunk is not a multiple of 3 bytes, then the chunk output will contain padding characters which should not exist in the middle of base64 data. These problems would be resolved if we had a way to convert encodings incrementally. The mbstring module appears to support incremental conversion under the hood, but it doesn't yet expose any incremental API to userland. Here's an example of how such an API might look: $context = mb_convert_init('UTF-8', 'UTF-16'); // To convert from UTF-16 to UTF-8. while (!$source->feof()) { $input_chunk = $source->read(8192); $output_chunk = mb_convert_add($context, $input_chunk, false); $dest->write($output_chunk); } $output_chunk = mb_convert_add($context, '', true); $dest->write($output_chunk); In the above example, the third argument of mb_convert_add() is set to true for the final chunk, to indicate that it should finalize the stream and flush any buffers. In usages that are structured more like stream filters, it may be more common for this to be called like "$output_chunk = mb_convert_add($this->context, $input_chunk, $closing);", where the final call may contain input data that should be added before finalizing the stream. -- Edit bug report at https://bugs.php.net/bug.php?id=80689&edit=1 -- Fix committed: https://bugs.php.net/fix.php?id=80689&r=fixed Fixed in release: https://bugs.php.net/fix.php?id=80689&r=alreadyfixed Need backtrace: https://bugs.php.net/fix.php?id=80689&r=needtrace Need Reproduce Script: https://bugs.php.net/fix.php?id=80689&r=needscript Try newer version: https://bugs.php.net/fix.php?id=80689&r=oldversion Not developer issue: https://bugs.php.net/fix.php?id=80689&r=support Expected behavior: https://bugs.php.net/fix.php?id=80689&r=notwrong Not enough info: https://bugs.php.net/fix.php?id=80689&r=notenoughinfo Submitted twice: https://bugs.php.net/fix.php?id=80689&r=submittedtwice register_globals: https://bugs.php.net/fix.php?id=80689&r=globals PHP version support discontinued: https://bugs.php.net/fix.php?id=80689&r=phptooold Daylight Savings: https://bugs.php.net/fix.php?id=80689&r=dst IIS Stability: https://bugs.php.net/fix.php?id=80689&r=isapi Install GNU Sed: https://bugs.php.net/fix.php?id=80689&r=gnused Floating point limitations: https://bugs.php.net/fix.php?id=80689&r=float No Zend Extensions: https://bugs.php.net/fix.php?id=80689&r=nozend MySQL Configuration Error: https://bugs.php.net/fix.php?id=80689&r=mysqlcfg