#46817 [NEW]: tokenizer misses last single-line comment

From: master dot jexus at gmail dot com Operating system: Windows XP SP3 PHP version: 5.3.0alpha3 PHP Bug Type: Unknown/Other Function Bug description: tokenizer misses last single-line comment Description: ------------ When using the tokenizer to lex given text, the output seems to miss the last token, if it was a single line comment. It only seems to occur if there isn't a newline behind the comment lexeme. Note the last entries in the arrays. Reproduce code: --------------- <?php print_r(token_get_all(file_get_contents(__FILE__))); // test $var = 5; // test Expected result: ---------------- Array ( [0] => Array ( [0] => 367 [1] => 1 ) [1] => Array ( [0] => 307 [1] => print_r [2] => 2 ) [2] => ( [3] => Array ( [0] => 307 [1] => token_get_all [2] => 2 ) [4] => ( [5] => Array ( [0] => 307 [1] => file_get_contents [2] => 2 ) [6] => ( [7] => Array ( [0] => 364 [1] => __FILE__ [2] => 2 ) [8] => ) [9] => ) [10] => ) [11] => ; [12] => Array ( [0] => 370 [1] => [2] => 2 ) [13] => Array ( [0] => 365 [1] => // test [2] => 4 ) [14] => Array ( [0] => 309 [1] => $var [2] => 5 ) [15] => Array ( [0] => 370 [1] => [2] => 5 ) [16] => = [17] => Array ( [0] => 370 [1] => [2] => 5 ) [18] => Array ( [0] => 305 [1] => 5 [2] => 5 ) [19] => ; [20] => Array ( [0] => 370 [1] => [2] => 5 ) [21] => Array ( [0] => 365 [1] => // test [2] => 6 ) ) Actual result: -------------- Array ( [0] => Array ( [0] => 368 [1] => 1 ) [1] => Array ( [0] => 307 [1] => print_r [2] => 2 ) [2] => ( [3] => Array ( [0] => 307 [1] => token_get_all [2] => 2 ) [4] => ( [5] => Array ( [0] => 307 [1] => file_get_contents [2] => 2 ) [6] => ( [7] => Array ( [0] => 365 [1] => __FILE__ [2] => 2 ) [8] => ) [9] => ) [10] => ) [11] => ; [12] => Array ( [0] => 371 [1] => [2] => 2 ) [13] => Array ( [0] => 366 [1] => // test [2] => 4 ) [14] => Array ( [0] => 309 [1] => $var [2] => 5 ) [15] => Array ( [0] => 371 [1] => [2] => 5 ) [16] => = [17] => Array ( [0] => 371 [1] => [2] => 5 ) [18] => Array ( [0] => 305 [1] => 5 [2] => 5 ) [19] => ; [20] => Array ( [0] => 371 [1] => [2] => 5 ) ) -- Edit bug report at http://bugs.php.net/?id=46817&edit=1 -- Try a CVS snapshot (PHP 5.2): http://bugs.php.net/fix.php?id=46817&r=trysnapshot52 Try a CVS snapshot (PHP 5.3): http://bugs.php.net/fix.php?id=46817&r=trysnapshot53 Try a CVS snapshot (PHP 6.0): http://bugs.php.net/fix.php?id=46817&r=trysnapshot60 Fixed in CVS: http://bugs.php.net/fix.php?id=46817&r=fixedcvs Fixed in CVS and need be documented: http://bugs.php.net/fix.php?id=46817&r=needdocs Fixed in release: http://bugs.php.net/fix.php?id=46817&r=alreadyfixed Need backtrace: http://bugs.php.net/fix.php?id=46817&r=needtrace Need Reproduce Script: http://bugs.php.net/fix.php?id=46817&r=needscript Try newer version: http://bugs.php.net/fix.php?id=46817&r=oldversion Not developer issue: http://bugs.php.net/fix.php?id=46817&r=support Expected behavior: http://bugs.php.net/fix.php?id=46817&r=notwrong Not enough info: http://bugs.php.net/fix.php?id=46817&r=notenoughinfo Submitted twice: http://bugs.php.net/fix.php?id=46817&r=submittedtwice register_globals: http://bugs.php.net/fix.php?id=46817&r=globals PHP 4 support discontinued: http://bugs.php.net/fix.php?id=46817&r=php4 Daylight Savings: http://bugs.php.net/fix.php?id=46817&r=dst IIS Stability: http://bugs.php.net/fix.php?id=46817&r=isapi Install GNU Sed: http://bugs.php.net/fix.php?id=46817&r=gnused Floating point limitations: http://bugs.php.net/fix.php?id=46817&r=float No Zend Extensions: http://bugs.php.net/fix.php?id=46817&r=nozend MySQL Configuration Error: http://bugs.php.net/fix.php?id=46817&r=mysqlcfg