Re: [RFC] Normalize token_get_all() output (with flag)

On Wed, Jan 6, 2016 at 1:43 AM, Nikita Popov <nikita.ppv@gmail.com> wrote: > Basically: token_get_all() is rather slow. I think it says something that > getting the tokens of a script is about as slow as lexing it, parsing it > into an internal AST and constructing an object-based userland AST for it. > If you use token_get_all() in a matter that only requires one lookahead > token at a time, you don't really care about how nice the token format is, > you're only interested in it being efficient. I was hoping that we can > optimize it by dropping the line numbers (which is the most volatile part of > the structure) and try to reuse the same array for tokens which have the > same ID and content (but likely different lineno). It's very likely that a > script contains the T_WHITESPACE( ) token more than one and similarly labels > and variables tend to repeat, etc. No idea if that would actually work/help, > just an idea. > Ah, I see what you mean. That'll take some thinking, and is well outside the scope of this RFC, but I'll give it some thought and maybe you or I or someone can gist something up later maybe... -Sara