#42396 [NEW]: Followup to #36711: __halt_compiler() and unicode detection

From: francois at tekwire dot net Operating system: all PHP version: 5.2.3 PHP Bug Type: Unicode Engine related Bug description: Followup to #36711: __halt_compiler() and unicode detection Description: ------------ Reopening bug #36711 because it is NOT a documentation problem. Setting 'detect_unicode=Off' is NOT a solution, just a workaround. In practice, because of this bug, PHK or PHAR packages cannot run on zend-multibyte-enabled environments, unless detect_unicode is turned off. Which makes them unusable in environments running unicode-encoded scripts. As a side effect, it also makes it impossible to include an unicode-encoded script inside a PHAR/PHK package, as it cannot be run. There is no logical reason to bind the __halt_compiler() feature with the zend-multibyte unicode detection capability. Everything after an __halt_compiler() directive must be considered as binary data and should not be scanned for unicode detection. If this data contains a unicode script, it will be scanned and detected when include()d through the stream wrapper. My (humble) suggestions to fix the problem: In zend_multibyte_detect_unicode(), the BOM detection does not have to be modified but, then, the script is scanned for null bytes : return zend_multibyte_detect_utf_encoding(LANG_SCNG(script_org), LANG_SCNG(script_org_size) TSRMLS_CC); There, the size should not be LANG_SCNG(script_org_size), but the offset of the __halt_compiler() directive. But I don't know where to find the COMPILER_HALT_OFFSET constant for the script. I even suspect it not to be available at this time... Another way, if the previous one is not possible, would be to scan for a binary string that cannot correspond to any unicode encoding. This way, PHK and PHAR could insert this string after ther __halt_compiler() directive, and it could be detected by zend_multibyte_detect_utf_encoding() as a stop string. I am ready to implement it if somebody provides a sequence of bytes that cannot be found in any unicode-encoded document. Reproduce code: --------------- <?php echo "OK\n"; __halt_compiler();<null-byte> Expected result: ---------------- OK Actual result: -------------- ?????????????????? -- Edit bug report at http://bugs.php.net/?id=42396&edit=1 -- Try a CVS snapshot (PHP 4.4): http://bugs.php.net/fix.php?id=42396&r=trysnapshot44 Try a CVS snapshot (PHP 5.2): http://bugs.php.net/fix.php?id=42396&r=trysnapshot52 Try a CVS snapshot (PHP 6.0): http://bugs.php.net/fix.php?id=42396&r=trysnapshot60 Fixed in CVS: http://bugs.php.net/fix.php?id=42396&r=fixedcvs Fixed in release: http://bugs.php.net/fix.php?id=42396&r=alreadyfixed Need backtrace: http://bugs.php.net/fix.php?id=42396&r=needtrace Need Reproduce Script: http://bugs.php.net/fix.php?id=42396&r=needscript Try newer version: http://bugs.php.net/fix.php?id=42396&r=oldversion Not developer issue: http://bugs.php.net/fix.php?id=42396&r=support Expected behavior: http://bugs.php.net/fix.php?id=42396&r=notwrong Not enough info: http://bugs.php.net/fix.php?id=42396&r=notenoughinfo Submitted twice: http://bugs.php.net/fix.php?id=42396&r=submittedtwice register_globals: http://bugs.php.net/fix.php?id=42396&r=globals PHP 3 support discontinued: http://bugs.php.net/fix.php?id=42396&r=php3 Daylight Savings: http://bugs.php.net/fix.php?id=42396&r=dst IIS Stability: http://bugs.php.net/fix.php?id=42396&r=isapi Install GNU Sed: http://bugs.php.net/fix.php?id=42396&r=gnused Floating point limitations: http://bugs.php.net/fix.php?id=42396&r=float No Zend Extensions: http://bugs.php.net/fix.php?id=42396&r=nozend MySQL Configuration Error: http://bugs.php.net/fix.php?id=42396&r=mysqlcfg