Re: [PATCH] Scanner "diet" with fixes, etc.

Hi Brian, ----- Original Message ----- From: "shire" Sent: Monday, May 04, 2009

Matt Wilmas wrote:

[...] How about this? #define SET_DOUBLE_QUOTES_SCANNED_LENGTH(len) CG(doc_comment_len) = (len) #define GET_DOUBLE_QUOTES_SCANNED_LENGTH() CG(doc_comment_len)

Sure, works for me ;-)

Cool. :-)

[...]

Have you considered using the lexer STATES and regex's instead of the manual C code for scanning the rest. It seems like if we have a one-char regex match for what the C code is doing we could handle this in the lexer without a lot of manual intervention (need to look at it more, just a thought I had earlier, the expressions are clearer now with your patch applied) ;-)

It seems that matching one-char-at-a-time with re2c would be more complicated than the manual way, not to mention slower than the old (current) way. Do you have any objection (well, you've kinda mentioned some :-)) if I'd commit the changes in a little while like Dmitry thought could be done?

Well I'm wondering if something more along these lines (just did this on-top of your patch as you cleaned up a lot) might be more appealing. (I'm not sure how much slower this would be than the current implementation, obviously it'll be somewhat slower, I'm basically just doing what you did in C but in the scanner instead of course). <ST_IN_SCRIPTING>"#"|"//" {

    BEGIN(ST_EOL_COMMENT);
    yymore();

} <ST_EOL_COMMENT>({NEWLINE}|"%>"|"?>") {

    char tmp = *(YYCURSOR-1);
    if ((tmp == '%' && CG(asp_tags)) | tmp == '?') {
      YYCURSOR -= 2;
    }
    CG(zend_lineno)++;
    BEGIN(ST_IN_SCRIPTING);
    return T_COMMENT;

} <ST_EOL_COMMENT>{ANY_CHAR} {

    if (YYCURSOR >= YYLIMIT) {
      BEGIN(ST_IN_SCRIPTING);
      return T_COMMENT;
    }
    yymore();

} Let me know what the thoughts are on the above, if we don't want that then I say yeah, commit away!

Wouldn't it be a little more complicated for strings/heredocs than comments? Or not, haven't thought about it much! :-) And you still need the "manual use" of YYCURSOR, etc. In other words, to me, the scanner rules are doing what the manual switch ()'s case statements do, but in a slower, "roundabout" way. Well, I'm gonna be away for a bit now, but I guess I can commit away when I get back.

-shire

- Matt