Bug #71207 [Opn->Nab]: token_get_all does not recognize context-sensitive keywords

Edit report at https://bugs.php.net/bug.php?id=71207&edit=1 ID: 71207 Updated by: nikic@php.net Reported by: kontakt at kukulich dot cz -Summary: token_get_all doesn't work correctly +Summary: token_get_all does not recognize context-sensitive keywords -Status: Open +Status: Not a bug Type: Bug Package: Unknown/Other Function PHP Version: 7.0.1 Block user comment: N Private report: N New Comment: If you want these to be recognized, you must pass the TOKEN_PARSE flag as the second argument to token_get_all(). Previous Comments: ------------------------------------------------------------------------ [2015-12-23 22:33:21] kontakt at kukulich dot cz Description: ------------ Thanks to https://wiki.php.net/rfc/context_sensitive_lexer it's possible in PHP7 to use a lot of keywords as constants, methods etc. However token_get_all doesn't return the right tokens. Test script: --------------- <?php $tokens = token_get_all('<?php class Test {const PUBLIC=0;}'); foreach ($tokens as $token) { if (is_array($token)) { echo token_name($token[0]) . "\n"; } } Expected result: ---------------- T_OPEN_TAG T_CLASS T_WHITESPACE T_STRING T_WHITESPACE T_CONST T_WHITESPACE T_STRING T_LNUMBER Actual result: -------------- T_OPEN_TAG T_CLASS T_WHITESPACE T_STRING T_WHITESPACE T_CONST T_WHITESPACE T_PUBLIC T_LNUMBER ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=71207&edit=1