Re: Where are we ACTUALLY on Unicode?

Rasmus Lerdorf wrote:

On 03/16/2010 12:05 PM, dreamcat four wrote:

On Tue, Mar 16, 2010 at 6:32 PM, Rasmus Lerdorf<rasmus@lerdorf.com> wrote:

On 03/16/2010 10:40 AM, dreamcat four wrote:

As for text files on disk, if they are unicode, they are most commonly utf-8 too. So then, why use utf-16 as internal unicode representation in Php? It doesn't really make a lot of sense for most regular people who want to use Php for their web application. Unless they don't really care how slow its gonna be converting everything, constantly...

Well, the obvious original reason is that ICU uses UTF-16 internally and the logic was that we would be going in and out of ICU to do all the various Unicode operations many more times than we would be interfacing with external things like MySQL or files on disk. You generally only read or write a string once from an external source, but you may perform multiple Unicode operations on that same string so avoiding a conversion for each operation seems logical. -Rasmus

Its only logical if you've bothered to profile the conversion calls to ICU against the non-ICU conversion calls. Im guessing the way to do that, is to have 2 versions of each conversion method. One used by ICU, and another used everywhere else. The harder part is to find some suitable, real life php programs to test with.

You mean check to see how many actual Unicode operations a standard app makes? We did talk about that, but there is a bit of a chicken-and-egg problem here. Because PHP doesn't natively support Unicode, people write apps in a way that lets them just pass Unicode through PHP and deal with it elsewhere. I would expect the profile to change once PHP gets better support for Unicode. But yes, some ideas around lazy conversions and other tricks would be interesting. If your input and output encoding are both utf-8 and all your data sources are utf-8 and you never do any sort of string manipulation on a particular string, why bother doing the utf-8 to utf-16 conversion on that string.

I think that is what I said originally ;) When a string is read in you set an extra flag if it needs special handling, otherwise you just handle it as a single byte per character string ... and for the diehards you add a switch to treat everything as it is now :) -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk// Firebird - http://www.firebirdsql.org/index.php