RE: Generic Tokenizer

Well, in the mean time I more or less finished the extension, and would like to submit it. So according to the manual I should make myself known on the list and say a few words about my extension. Presumably so that I can then be properly flamed... So a few words about myself are that I have been working with PHP for 7 years now. It's my whole IT career. Currently employed as a technical team lead in London, although I'm originally from the same place as Zend.. :-) The extension, as I stated in my previous email is very simple. The tokenizer is a very basic "general purpose" tokenizer that will break down a string into tokens of simple types: alphanumeric, whitespace, parentheses, punctuation and non-printing. It is not mean to be very flexible and configurable, but is only meant to serve as the base of a user-level parser. The name of the extension is 'gtoken'. The following is a sample core and its corresponding output: <?php $tokens = gtoken_tokenize(" SELECT Email.*, User.name FROM Email (LEFT JOIN User ON (Email.from = User.username)) ORDER BY (Email.received) ", GTOKEN_TOKENIZE_DISCARD_WS); foreach ( $tokens as $token ) { echo str_pad($token['token'], 30), gtoken_token_name($token['type']), "\n"; } ?> SELECT GTOKEN_TOKEN_ALPHANUMERIC Email GTOKEN_TOKEN_ALPHANUMERIC .*, GTOKEN_TOKEN_PUNCTUATION User GTOKEN_TOKEN_ALPHANUMERIC . GTOKEN_TOKEN_PUNCTUATION name GTOKEN_TOKEN_ALPHANUMERIC FROM GTOKEN_TOKEN_ALPHANUMERIC Email GTOKEN_TOKEN_ALPHANUMERIC ( GTOKEN_TOKEN_PARENTHESIS LEFT GTOKEN_TOKEN_ALPHANUMERIC JOIN GTOKEN_TOKEN_ALPHANUMERIC User GTOKEN_TOKEN_ALPHANUMERIC ON GTOKEN_TOKEN_ALPHANUMERIC ( GTOKEN_TOKEN_PARENTHESIS Email GTOKEN_TOKEN_ALPHANUMERIC . GTOKEN_TOKEN_PUNCTUATION from GTOKEN_TOKEN_ALPHANUMERIC = GTOKEN_TOKEN_PUNCTUATION User GTOKEN_TOKEN_ALPHANUMERIC . GTOKEN_TOKEN_PUNCTUATION username GTOKEN_TOKEN_ALPHANUMERIC ) GTOKEN_TOKEN_PARENTHESIS ) GTOKEN_TOKEN_PARENTHESIS ORDER GTOKEN_TOKEN_ALPHANUMERIC BY GTOKEN_TOKEN_ALPHANUMERIC ( GTOKEN_TOKEN_PARENTHESIS Email GTOKEN_TOKEN_ALPHANUMERIC . GTOKEN_TOKEN_PUNCTUATION received GTOKEN_TOKEN_ALPHANUMERIC ) GTOKEN_TOKEN_PARENTHESIS