Re: RC2 and integer/float handling in 5.3

Hey Matt, Matt Wilmas wrote:

Yep, 5.3's snapshot self-compiled from a couple days ago on Windows (not that that should matter). (I'm not regenerating it with re2c, which also shouldn't matter; using the existing .c file. I haven't touched the scanner stuff in a long time (yet) to regen.) Scanner of course hasn't changed since then.

Here's what I'm currently doing (more or less with some changed paths): $ echo $PHPCVS :pserver:shire@cvs.php.net:/repository $ cvs -d $PHPCVS co -r PHP_5_3 php5 ... $ cd php5 $ ./buildconf && ../config.nice && make ... $ cat ../bug46817-1.php <?php var_dump(token_get_all(file_get_contents('../bug46817-2.php'))); $ cat ../bug46817-2.php <?php // this comment and trailing blank contain windows CR+LF $ ./sapi/cli/php ../bug46817-1.php array(3) { [0]=> array(3) {

    [0]=>
    int(368)
    [1]=>
    string(6) "<?php

"

    [2]=>
    int(1)

} [1]=> array(3) {

    [0]=>
    int(366)
    [1]=>

" string(57) "// this comment and trailing blank contain windows CR+LF

    [2]=>
    int(2)

} [2]=> array(3) {

    [0]=>
    int(371)
    [1]=>
    string(3) "

"

    [2]=>
    int(2)

} } The newlines look like this in the second file: <?php$ // this comment and trailing blank contain windows CR+LF^M$ ^M$ Unfortunately I can't test on a windows build, perhaps you could re-test or share your reproduction that fails as this seems to work for me unless I'm of course missing some difference.

Test case is the one in the bug report. :-) Last token is not the comment, but whitespace.

There are two reproductions in the bug report ;-)

Also, the unterminated comment Warning is still missing with "<?php /* blah " like it's been since the re2c change (except maybe for the time your fix was applied). My changes would clean this up of course, unless you do something first.

I think fixing this would be great as well as the other highlighter test that was changed. I would just prefer that the scanner handle these rather than us implementing what is essentially a hand-written scanner within the lexer file. -shire