Bug #69430 [Asn->Csd]: token_get_all has new irrecoverable errors

Edit report at https://bugs.php.net/bug.php?id=69430&edit=1 ID: 69430 Updated by: nikic@php.net Reported by: lauri dot kentta+php-bugs at gmail dot com Summary: token_get_all has new irrecoverable errors -Status: Assigned +Status: Closed Type: Bug Package: Scripting Engine problem PHP Version: master-Git-2015-04-12 (Git) Assigned To: nikic Block user comment: N Private report: N New Comment: Automatic comment on behalf of nikic Revision: http://git.php.net/?p=php-src.git;a=commit;h=d91aad5966f01259f0e1a431a754d917807761b5 Log: Fix bug #69430 Previous Comments: ------------------------------------------------------------------------ [2015-04-12 10:36:55] nikic@php.net Good point. I changed these fatals to ParseExceptions to make it possible to handle them gracefully at all, but you're right that throwing from token_get_all() will be a problem as well for some usages. I'd probably go with returning E_ERROR tokens in this case, as we already do it internally. It would be good to still be able to access the exceptions that are thrown in some way. ------------------------------------------------------------------------ [2015-04-12 10:28:10] lauri dot kentta+php-bugs at gmail dot com Description: ------------ Previously, token_get_all would handle even invalid input. This is no longer the case, since some people added unrecoverable errors to the lexer (lex_scan) for unicode escapes and invalid octal literals. This means that token_get_all can't be used for custom highlighting anymore, because invalid scripts won't produce any output. Interestingly highlight_string is still working, but I couldn't see why. The commits that probably cause the problem: commit bae46f307c2d0cdef9b8f5426adcc46920776700 Author: Andrea Faulds <ajf@ajf.me> Date: Fri Dec 19 00:40:59 2014 +0000 Unicode Codepoint Escape Syntax commit 5f29b980514867f1a09969ca6a1c1f5fb00c3027 Author: Andrea Faulds <ajf@ajf.me> Date: Fri Jan 9 06:32:36 2015 +0000 Error on invalid octal (fixes PHPSadness #31) commit a8bf1c5d8f5755b53492e58040cfe88150eb57b6 Author: Nikita Popov <nikic@php.net> Date: Sat Mar 21 20:10:19 2015 +0100 Throw ParseException from lexer Test script: --------------- <?php var_dump(token_get_all('<?php 09 "\u{bar}"')); Expected result: ---------------- Some meaningful tokens such as: T_OPEN_TAG <?php T_LNUMBER 09 T_CONSTANT_ENCAPSED_STRING "\u{bar}" or T_OPEN_TAG <?php T_LNUMBER_INVALID 09 T_CONSTANT_ENCAPSED_STRING_INVALID "\u{bar}" or even T_OPEN_TAG <?php T_ERROR 09 T_ERROR "\u{bar}" Actual result: -------------- ParseException: Invalid numeric literal ParseException: Invalid UTF-8 codepoint escape sequence ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=69430&edit=1