Re: [PATCH] Scanner "diet" with fixes, etc.

On Mon, May 4, 2009 at 5:51 PM, shire <shire@php.net> wrote: > Arnaud Le Blanc wrote: >> >> Hi, >> On Mon, May 4, 2009 at 9:36 AM, shire<shire@php.net>  wrote: >>> >>> Regarding the ZEND_MMAP_AHEAD issue and the temp. fix that Dmitry put in >>> we >>> need to find a solution to that, perhaps I can play with that this week >>> too >>> as I think I'm seeing some related issues in my testing of 5.3. >>>  Essentially >>> we abuse ZEND_MMAP_AHEAD by adding it to the file size and passing it to >>> the >>> mmap call which isn't at all valid and only really works up to PAGESIZE.. >>>  We >>> could possibly use YYFILL to re-allocate more space as necessary past the >>> end of file to fix this. >> >> I was thinking of doing something like that with YYFILL too. However >> there is a bunch of pointers to take in to account and to update (e.g. >> yy_marker, yy_text, etc). >> > > Yeah, I'm pretty sure that's how most of the example re2c code is setup: > > #define YYFILL(n) {cursor = fill(s, cursor);} > > uchar *fill(Scanner *s, uchar *cursor){ >    if(!s->eof){ >  unint cnt = s->lim - s->tok; >  uchar *buf = malloc((cnt + 1)*sizeof(uchar)); >  memcpy(buf, s->tok, cnt); >  cursor = &buf[cursor - s->tok]; >  s->pos = &buf[s->pos - s->tok]; >  s->ptr = &buf[s->ptr - s->tok]; >  s->lim = &buf[cnt]; >  s->eof = s->lim; *(s->eof)++ = '\n'; >  s->tok = buf; >    } >    return cursor; > } > > > > -shire > This is what I seen too, but this is not always applicable. The scanner have code that refers to yy_text, yy_start, yy_cursor, yy_marker, etc. All those pointers point to the original buffer and must be updated by fill(). At each point in time the scanner may rollback to yy_marker or a rule may want to fetch yy_text or yy_start at any time. So the buffer must be large enough to contain all data from min(all_of_them) to max(all_of_them). That makes things a little complicated and potentially less efficient than a big buffer for the whole file. Regards, Arnaud