Re: [RFC] Replace the flex-based scanner with an re2c [1] based lexer

Hi!

Were the stream support issues solved?

We completely dropped multibyte support. The reason is that the way we were

I wasn't asking about multibyte (that we discuss below), but about other streams - I think I mentioned it on IRC last time re2c parser was discussed. I remember re2c used mmap, and not all files PHP can run can be mmap'ed. Was it fixed?

Once we have finished the move to re2c, we can support all of those correctly. The multibyte support also duplicated the encoding tables otherwise available in ext/mbstring or ext/iconv or pecl/intl.

pecl/intl per se doesn't have any encoding tables. ICU does, but that would mean you have to have ICU to run PHP. That might not be a big problem since ICU is supported by IBM (read: good chance more "exotic" systems would have support) it is still dependency on non-bundled 3rd party library in PHP 5 core. Of course, PHP 6 has this dependency, but we might want to not have such things in 5.x so that you won't have to change your system too much while staying on 5.x.

Rely on a not supported undocumented feature? I am rather able to build php and rewrite that support.

Being undocumented is nothing to be proud of, however as poorly documented as it is, it is used. I'm all for implementing it in a better way - and having new parser is a good time to do it. That's exactly the reason we shouldn't rush with it but do it right this time. There's no burning need to have a new parser right now, so we can have some moment to think - ok, how we want multibyte support there to work? And if we might need some modifications, we'd have time and flexibility to do it, not having the code in 5.3 which was supposed to go in RC in Q1 (ending 1 month from now).

You are free to contribute and make MB support working upfront.

I know I'm free :) However, as much as I understand the eagerness of having it in the source tree, I repeat that I do not think dropping multibyte support in 5.3 is acceptable. Thus, if it is committed right now, 5.3 would have to be deferred until this is resolved. If this is resolved timely for 5.3 - great. If not, we better get it in 5.4 right than in 5.3 wrong. -- Stanislav Malyshev, Zend Software Architect stas@zend.com http://www.zend.com/ (408)253-8829 MSN: stas@zend.com