Bug #71207 [NEW]: token_get_all doesn't work correctly

From: kontakt at kukulich dot cz Operating system: PHP version: 7.0.1 Package: Unknown/Other Function Bug Type: Bug Bug description:token_get_all doesn't work correctly Description: ------------ Thanks to https://wiki.php.net/rfc/context_sensitive_lexer it's possible in PHP7 to use a lot of keywords as constants, methods etc. However token_get_all doesn't return the right tokens. Test script: --------------- <?php $tokens = token_get_all('<?php class Test {const PUBLIC=0;}'); foreach ($tokens as $token) { if (is_array($token)) { echo token_name($token[0]) . "\n"; } } Expected result: ---------------- T_OPEN_TAG T_CLASS T_WHITESPACE T_STRING T_WHITESPACE T_CONST T_WHITESPACE T_STRING T_LNUMBER Actual result: -------------- T_OPEN_TAG T_CLASS T_WHITESPACE T_STRING T_WHITESPACE T_CONST T_WHITESPACE T_PUBLIC T_LNUMBER -- Edit bug report at https://bugs.php.net/bug.php?id=71207&edit=1 -- Try a snapshot (PHP 5.4): https://bugs.php.net/fix.php?id=71207&r=trysnapshot54 Try a snapshot (PHP 5.5): https://bugs.php.net/fix.php?id=71207&r=trysnapshot55 Try a snapshot (trunk): https://bugs.php.net/fix.php?id=71207&r=trysnapshottrunk Fixed in SVN: https://bugs.php.net/fix.php?id=71207&r=fixed Fixed in release: https://bugs.php.net/fix.php?id=71207&r=alreadyfixed Need backtrace: https://bugs.php.net/fix.php?id=71207&r=needtrace Need Reproduce Script: https://bugs.php.net/fix.php?id=71207&r=needscript Try newer version: https://bugs.php.net/fix.php?id=71207&r=oldversion Not developer issue: https://bugs.php.net/fix.php?id=71207&r=support Expected behavior: https://bugs.php.net/fix.php?id=71207&r=notwrong Not enough info: https://bugs.php.net/fix.php?id=71207&r=notenoughinfo Submitted twice: https://bugs.php.net/fix.php?id=71207&r=submittedtwice register_globals: https://bugs.php.net/fix.php?id=71207&r=globals PHP 4 support discontinued: https://bugs.php.net/fix.php?id=71207&r=php4 Daylight Savings: https://bugs.php.net/fix.php?id=71207&r=dst IIS Stability: https://bugs.php.net/fix.php?id=71207&r=isapi Install GNU Sed: https://bugs.php.net/fix.php?id=71207&r=gnused Floating point limitations: https://bugs.php.net/fix.php?id=71207&r=float No Zend Extensions: https://bugs.php.net/fix.php?id=71207&r=nozend MySQL Configuration Error: https://bugs.php.net/fix.php?id=71207&r=mysqlcfg