Re: 5.3 items

Hi Brian, ----- Original Message ----- From: "shire" Sent: Monday, March 09, 2009

Hey Matt, Matt Wilmas wrote:

9. tokenizer misses last single-line comment (http://bugs.php.net/bug.php?id=46817)

I was going to take care of that one, as I mentioned in a previous message, though it's been awhile since I've been delayed much longer with stuff here. :-( (Nothing set up for building PHP on this system yet; hope to in the next several hours finally, and do some things!)

Sorry I missed you're earlier email. I saw this sitting on the 5.3 todo list and it was breaking some of our parsing so I figured I'd take a stab at it. Here is my current patch http://tekrat.com/downloads/bits/php53.scanner_eof.patch, please let me know if you have some suggestions/changes. It sounds like you commented on this initially so please let me know what you/we should do ie: merging my patch/your work, commiting this, or if you had a better fix in mind etc. My biggest complaint is that my current patch requires adding \x00 to any exclusion rules ("[^"). These changes for handling EOF should probably be ported to the INI scanner as well for the above reason and to keep them similar.

I don't have much time right now, but looked at it quick, and see that you're actually trying to work around the re2c issues in general. :-) I was only thinking of putting a "band-aid" on the comment symptom(s), since those are about the only ones that occur with valid code (is the tokenizer ext. *supposed to* handle all tokens in code that wouldn't really compile?). And yeah, about excluding \x00 from ANY_CHAR, it could change things, since it's always been allowed, although it seems strange that code would have literal NULLs in it (generated eval()'d code?). That was part of the reason I couldn't come up with a generic fix while keeping all behavior. If re2c would just remember the last matching state it was in at EOF like Flex! Otherwise, I don't know what to do. :-/ I'm going to do something else before trying to implement what I was going to do, so there's no patch yet...

As far as I know there's still the other comment-related issue where no Warning is giving about "Unterminated comment ..." for unclosed /* ... It's all of course related to the fundamental re2c issue, for now, where when the scanned input ends while a variable length part of a rule is being matched, it just aborts ("return 0;") in YYFILL().

I don't seem to see this problem, perhaps I'm not reproducing it correctly?

As far as the Warning, with "<?php /* blah " do you get "Unterminated comment ..." ? Of course your patch would restore it, because it's missing last I checked (not able to right now).

And that applies to the case Lukas gave in the bug report: WHITESPACE pattern is variable length.

Didn't see/find this is there a bug # or link?

I meant the "could be related if not the same problem" comment added the other day in Bug #46817.

-shire

- Matt