[PEPr] Comment on PHP::Lexer

Harry Fuecks (http://pear.php.net/user/hfuecks) has commented on the proposal for PHP::Lexer. Comment: Looks cool. I know the original Lexer from SimpleTest quite well now, from building a wiki parser based on it. Think the approach you've used, returning the matches as an array (vs. callbacks to functions) will make this more appealing to many plus deliver better performance. At the same time, it might be worth introducing the "modes" that were in the original Lexer - these basically solve the problem of state machines - you might have something like; // 'base' in the initial state $Grammar->addEntryRule('base','pre','<pre>'); $Grammar->addExitRule('pre','</pre>'); Now no other rules will be applied while inside the <pre /> tags Does it support users providing regexes that contain parenthesis? I wonder because the version I know escapes these, to disable subpatterns and prevent conflicts with the internal regex subpatterns (had to hack it so I could use lookbehinds / lookaheads and setting regex options from inside the expression). Also, would it be worth having an alternative UTF-8 version of the Lexer? Personally have found it's doable using the /u pattern modifier in all the pcre calls as well as having alternative implementations of some of the str fns. Have a few you're welcome to re-use here: http://cvs.sourceforge.net/viewcvs.py/xmlrpccom/dokuwiki/lib/utf8_string.php?rev=1.5 - from the looks of it that would only leave you needing to reimplement substr_count() Following Alan's suggestion on the package name, what about Text_Lexer_Regex? Could imagine there might be a Text_Lexer_Str, based on the string functions, someday. Proposal information: http://pear.php.net/pepr/pepr-proposal-show.php?id=197 -- Sent by PEPr, the automatic proposal system at http://pear.php.net