Re: [ZEND-ENGINE-CVS] cvs: ZendEngine2(PHP_5_3) / Zend.m4 zend.c zend_compile.c zend_globals.h zend_highlight.c zend_language_scanner.c
zend_language_scanner.h zend_language_scanner.l zend_language_scanner_defs.h zend_multibyte.h

Hello Rui, Sunday, July 6, 2008, 3:31:40 AM, you wrote: > Hello Marcus, > On Sat, 5 Jul 2008 20:46:33 +0200 > Marcus Boerger <helly@php.net> wrote: >> Hello Rui, >> >> briefly scanning to this you took a complete different approach I did. >> While used newer re2c features to generate two different scanners you seem >> to have reapplied the the old code. That would mean you're most likely a >> bit slower but have less code too. >> Did you change anything or did you >> simply reapply one by one? If you did just copy I might spend some time >> appliying a few fixes I encountered. > Thank you for comments. > My implementation is ported code from PHP 5.2 with minimal change > because I am not familiar with lexical analizer including re2c, flex. > I read man page of re2c on the web site, and Peter's paper, > but, I couldn't understand the detail. > What we need to define for multibyte encoding support is > how to treat some flex incompatible encodings like Shift_JIS. > The implementation is like, > 1. read script from file system. > 2. convert from script encoding to flex compatible encoding > 3. applied to lexer > 4.convert from flex compatible encoding to internal encoding (if > necessary) > The script encoding is specified by mbstinrg.scirpt_encoding > in php.ini , the script file itself as BOM in Unicode (like UTF-8, UTF16), > in the script file using 'declare'. > The approach used in PHP 5.2 might be ineffective for re2c. > I will appreciate if you make a better implementaion based on > the advanced feature of re2c. >> Also we are still missing tests. >> Without tests I feel we have to remove all of this anyway. > Currently, a couple of test scripts is already existing in > Zend/tests/multibyte/*.phpt > 'still missing' means we should make a huge number of test scripts > or a higher percentage of code coverage is necessary ? > The encoding converstion engine part is already tested > as mbstring extension. > The scanner part including encoding conversion, encoding > detection should be tested. > The current test scipts are covering the most important > feature of zend-multibyte support, > 1. test for flex incompatible encoding (0x5c issue) and 'declare' sentence > multibye_encoding_001.phpt We don't have a test yet that switches encodings with multiple declare statements or should that be disallowed? > 2. test for UTF-8 BOM > multibye_encoding_002.phpt > 3. test for UTF-16 BOM and UTF-16 encoded Unicode script > multibye_encoding_003.phpt > The encoding conversion feature is also covered by these scripts. > What kind of test scripts do you need ? Anything that brings code coverage for the multibyte stuff as high as possible. Best regards, Marcus