Bug #76437 [Ver]: token_get_all with TOKEN_PARSE flag fails to recognise close tag

Edit report at https://bugs.php.net/bug.php?id=76437&edit=1 ID: 76437 Updated by: cmb@php.net Reported by: nathanielzimmermann at gmail dot com Summary: token_get_all with TOKEN_PARSE flag fails to recognise close tag Status: Verified Type: Bug Package: Scripting Engine problem PHP Version: 7.3.0alpha1 Assigned To: dmitry Block user comment: N Private report: N New Comment: Thanks! A small improvement might be to replace the integer literals with sizeof("?>")-1 and sizeof("<?=")-1, respectively. Previous Comments: ------------------------------------------------------------------------ [2018-06-17 14:32:55] laruence@php.net I am afraid that the special cases should be handled in tokenizer side now :< diff --git a/ext/tokenizer/tokenizer.c b/ext/tokenizer/tokenizer.c index ef9d136..9524c69 100644 --- a/ext/tokenizer/tokenizer.c +++ b/ext/tokenizer/tokenizer.c @@ -191,8 +191,16 @@ void on_event(zend_php_scanner_event event, int token, int line, void *context) switch (event) { case ON_TOKEN: - if (token == END) break; - add_token(token_stream, token, LANG_SCNG(yy_text), LANG_SCNG(yy_leng), line); + { + if (token == END) break; + /* Specical cases */ + if (token == ';' && LANG_SCNG(yy_leng) == 2) { + token = T_CLOSE_TAG; + } else if (token == T_ECHO && LANG_SCNG(yy_leng) == 3) { + token = T_OPEN_TAG_WITH_ECHO; + } + add_token(token_stream, token, LANG_SCNG(yy_text), LANG_SCNG(yy_leng), line); + } break; case ON_FEEDBACK: tokens_ht = Z_ARRVAL_P(token_stream); ------------------------------------------------------------------------ [2018-06-09 13:38:27] cmb@php.net It seems that this behavioral change is an unintended side effect of commit 8afb91c[1]. Dmitry, could you have a look at this issue, please? [1] <http://git.php.net/?p=php-src.git;a=commit;h=8afb91cdadaa5a40ee972dec278d89c76bf454e3> ------------------------------------------------------------------------ [2018-06-09 11:26:16] nathanielzimmermann at gmail dot com Description: ------------ token_get_all with the TOKEN_PARSE flag fails to recognise the T_CLOSE_TAG token Test script: --------------- <?php var_dump(token_get_all('<?php echo 1; ?>', TOKEN_PARSE)[6]); Expected result: ---------------- array(3) { [0]=> int(381) [1]=> string(2) "?>" [2]=> int(1) } Actual result: -------------- string(2) "?>" ------------------------------------------------------------------------ -- Edit this bug report at https://bugs.php.net/bug.php?id=76437&edit=1