Internationalization

Hi (here comes a lengthy mail!) I have discussed this issue with Rasmus privately as well as on php-dev list couple times while ago. Thanks to both sgk@happysize.co.jp and tsukada@fminn.nagano.nagano.jp, I18N version is fairly complete. Since it's becoming much more painfull to keeping up with main distribution, we are hoping to have the i18n part incorporated into the main distribution. Here are some background on what it does (if you don't care about the nitty gritty detail, skip this): ---------------------------------------------------------------------- (the main concern is to handle Japanese, but many other languages faces common situation with having multiple character encodings for same language) o Problem with current distribution When dealing with Japanese, you have to accommadate multiple character codes, mainly SJIS, EUC, JIS, and UTF-8. To make long story short, to handle Japanese properly, PHP needs to accept any type of encoding and convert the codes to desired internal encoding. At the sametime, the output code should be controlled as well. There are few other modifications needed to accommadate Japanese as well: mail and regex. o I18N version Here are the list of changes in rough cut - added conversion filters (as we call it) for SJIS, EUC, JIS and UTF-8 - YY_INPUT in language-scanner is overloaded to include filtering process - http output is filtered - mail is modified to comply with RFC for sending email in Japanese - php3.ini addition to configure how PHP handles various codes - should be easy to implement other language - to enable Japanese support, one should compile it with --enable-i18n o How common users uses PHP In php3.ini, a user may set the encoding scheme as such: i18n.script_encoding = AUTO i18n.http_input = AUTO i18n.internal_encoding = EUC i18n.http_output = SJIS This means, php3 script can be in any encoding, as well as any post or get data. They will be converted to EUC automagicaly. Now, since the internal encoding is in EUC, saving any data to MySQL (for an example) will be in EUC. This is perfectly fine since MySQL will only handle either SJIS or EUC. Samething applies with any other database, output encoding is whatever the internal encoding is. At last, output is set to SJIS as default, but can be controlled via php function. o Todo Regex! This is the most painfull part. Since we allow EUC and SJIS to be used as internal encoding, regex should be written to accommadate both. This sucks. The best thing is to find (if any) regex that can handle unicode, but this means we need to use UTF-8 as internal encoding. This is where it gets tricky, if internal encoding is UTF, then any output other than regular http output will be in UTF. We suspect that many applications that interact with PHP can't handle UTF. The only way to get around this is to place filtering at each output process (that's alot of work for every single developers). Any ideas on this? anyone? Can't compile under Windows. Looking to get more developers in this area. ---------------------------------------------------------------------- We are hoping that by merging with main distribution, we will gain more Japanese supporters. AFAIK, there is one Japanese book already in publish that deals with PHP. Core PHP Programming is in process of translation. So this should be damn good timing. This is more than Japanese hack for PHP. We spent long time discussing the implementation with ultimate goal to be true i18n support. Any multi-byte user can look at the code and should be able to add their language support. I think the unicode part should be very interesting to look at. I know sgk has CVS access, so he may be able to commit the changes if he has time. If not I will give it a try. Any comments would be greately appreciated! Hiro //--- Hironori Sato --------------------------------------- KB9HAD --- // satoh@jpnnet.com http://staff.jpnnet.com/satoh/ // Japanese Network http://www.jpnnet.com/