Bug #69430 [Opn]: token_get_all has new irrecoverable errors

Edit report at https://bugs.php.net/bug.php?id=69430&edit=1 ID: 69430 Updated by: nikic@php.net Reported by: lauri dot kentta+php-bugs at gmail dot com Summary: token_get_all has new irrecoverable errors Status: Open Type: Bug Package: Scripting Engine problem PHP Version: master-Git-2015-04-12 (Git) -Assigned To: +Assigned To: nikic Block user comment: N Private report: N New Comment: Good point. I changed these fatals to ParseExceptions to make it possible to handle them gracefully at all, but you're right that throwing from token_get_all() will be a problem as well for some usages. I'd probably go with returning E_ERROR tokens in this case, as we already do it internally. It would be good to still be able to access the exceptions that are thrown in some way. Previous Comments: ------------------------------------------------------------------------ [2015-04-12 10:28:10] lauri dot kentta+php-bugs at gmail dot com Description: ------------ Previously, token_get_all would handle even invalid input. This is no longer the case, since some people added unrecoverable errors to the lexer (lex_scan) for unicode escapes and invalid octal literals. This means that token_get_all can't be used for custom highlighting anymore, because invalid scripts won't produce any output. Interestingly highlight_string is still working, but I couldn't see why. The commits that probably cause the problem: commit bae46f307c2d0cdef9b8f5426adcc46920776700 Author: Andrea Faulds <ajf@ajf.me> Date: Fri Dec 19 00:40:59 2014 +0000 Unicode Codepoint Escape Syntax commit 5f29b980514867f1a09969ca6a1c1f5fb00c3027 Author: Andrea Faulds <ajf@ajf.me> Date: Fri Jan 9 06:32:36 2015 +0000 Error on invalid octal (fixes PHPSadness #31) commit a8bf1c5d8f5755b53492e58040cfe88150eb57b6 Author: Nikita Popov <nikic@php.net> Date: Sat Mar 21 20:10:19 2015 +0100 Throw ParseException from lexer Test script: --------------- <?php var_dump(token_get_all('<?php 09 "\u{bar}"')); Expected result: ---------------- Some meaningful tokens such as: T_OPEN_TAG <?php T_LNUMBER 09 T_CONSTANT_ENCAPSED_STRING "\u{bar}" or T_OPEN_TAG <?php T_LNUMBER_INVALID 09 T_CONSTANT_ENCAPSED_STRING_INVALID "\u{bar}" or even T_OPEN_TAG <?php T_ERROR 09 T_ERROR "\u{bar}" Actual result: -------------- ParseException: Invalid numeric literal ParseException: Invalid UTF-8 codepoint escape sequence ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=69430&edit=1