RE: [PECL-DEV] RE: Generic Tokenizer

> From: Antony Dovgal > > So what is this and why is it useful? I explained it in a previous email but I'll happily re-iterate: The extension is a general-purpose tokenizer that breaks down a given string to general tokens: alphanumeric tokens, whitespace, strings and punctuation. It returns an array of tokens. It is useful because in a lot of PHP frameworks, tools, libraries etc, parsing of strings takes place. These strings often represent all sorts of small languages (OQL for example) or formats. The user-space parsers of these languages are usually heavily reliant on regexps, not very efficient, and each implements the tokenizing process individually. This extension is meant to both take away the need to build the basic tokenizer and also gives a performance boost in that area as it's implemented in C rather then in user-space code. It is meant to serve as the basis for such user-space parsers. The user-space parser is only left with the job of lexically processing an array of tokens. I've added an example: http://gtoken.abiri.org/example.source.php And this is its output: http://gtoken.abiri.org/example.php A