Bug #69430 [NEW]: token_get_all has new irrecoverable errors

From: lauri dot kentta+php-bugs at gmail dot com Operating system: PHP version: master-Git-2015-04-12 (Git) Package: Scripting Engine problem Bug Type: Bug Bug description:token_get_all has new irrecoverable errors Description: ------------ Previously, token_get_all would handle even invalid input. This is no longer the case, since some people added unrecoverable errors to the lexer (lex_scan) for unicode escapes and invalid octal literals. This means that token_get_all can't be used for custom highlighting anymore, because invalid scripts won't produce any output. Interestingly highlight_string is still working, but I couldn't see why. The commits that probably cause the problem: commit bae46f307c2d0cdef9b8f5426adcc46920776700 Author: Andrea Faulds <ajf@ajf.me> Date: Fri Dec 19 00:40:59 2014 +0000 Unicode Codepoint Escape Syntax commit 5f29b980514867f1a09969ca6a1c1f5fb00c3027 Author: Andrea Faulds <ajf@ajf.me> Date: Fri Jan 9 06:32:36 2015 +0000 Error on invalid octal (fixes PHPSadness #31) commit a8bf1c5d8f5755b53492e58040cfe88150eb57b6 Author: Nikita Popov <nikic@php.net> Date: Sat Mar 21 20:10:19 2015 +0100 Throw ParseException from lexer Test script: --------------- <?php var_dump(token_get_all('<?php 09 "\u{bar}"')); Expected result: ---------------- Some meaningful tokens such as: T_OPEN_TAG <?php T_LNUMBER 09 T_CONSTANT_ENCAPSED_STRING "\u{bar}" or T_OPEN_TAG <?php T_LNUMBER_INVALID 09 T_CONSTANT_ENCAPSED_STRING_INVALID "\u{bar}" or even T_OPEN_TAG <?php T_ERROR 09 T_ERROR "\u{bar}" Actual result: -------------- ParseException: Invalid numeric literal ParseException: Invalid UTF-8 codepoint escape sequence -- Edit bug report at https://bugs.php.net/bug.php?id=69430&edit=1 -- Try a snapshot (PHP 5.4): https://bugs.php.net/fix.php?id=69430&r=trysnapshot54 Try a snapshot (PHP 5.5): https://bugs.php.net/fix.php?id=69430&r=trysnapshot55 Try a snapshot (trunk): https://bugs.php.net/fix.php?id=69430&r=trysnapshottrunk Fixed in SVN: https://bugs.php.net/fix.php?id=69430&r=fixed Fixed in release: https://bugs.php.net/fix.php?id=69430&r=alreadyfixed Need backtrace: https://bugs.php.net/fix.php?id=69430&r=needtrace Need Reproduce Script: https://bugs.php.net/fix.php?id=69430&r=needscript Try newer version: https://bugs.php.net/fix.php?id=69430&r=oldversion Not developer issue: https://bugs.php.net/fix.php?id=69430&r=support Expected behavior: https://bugs.php.net/fix.php?id=69430&r=notwrong Not enough info: https://bugs.php.net/fix.php?id=69430&r=notenoughinfo Submitted twice: https://bugs.php.net/fix.php?id=69430&r=submittedtwice register_globals: https://bugs.php.net/fix.php?id=69430&r=globals PHP 4 support discontinued: https://bugs.php.net/fix.php?id=69430&r=php4 Daylight Savings: https://bugs.php.net/fix.php?id=69430&r=dst IIS Stability: https://bugs.php.net/fix.php?id=69430&r=isapi Install GNU Sed: https://bugs.php.net/fix.php?id=69430&r=gnused Floating point limitations: https://bugs.php.net/fix.php?id=69430&r=float No Zend Extensions: https://bugs.php.net/fix.php?id=69430&r=nozend MySQL Configuration Error: https://bugs.php.net/fix.php?id=69430&r=mysqlcfg