Re: What "Unicode support" really means

On Wed, Mar 5, 2014 at 11:37 PM, Crypto Compress <cryptocompress@googlemail.com> wrote: > Am 05.03.2014 23:04, schrieb Lester Caine: > >> Andrea Faulds wrote: >>> >>> I’m unconvinced that dealing with printed characters instead of >>> codepoints is a good idea generally. It’s necessary when doing certain types >>> of text manipulation, but I don’t think we should do this for substr() and >>> general string manipulation. You can’t always necessarily split a group code >>> points into distinct characters, where a code point joins two characters >>> together visually, for example. >> >> >> I've flagged up before that there are at least three string lengths. >> 1 - Number of bytes >> 2 - Number of codepoints >> 3 - Number of glyphs >> >> But the answer to those figures also depend on other factors :( >> >> Initially I thought that for string manipulation switching to UTF32 was a >> no-brainer, but even that has problems when a code point can actually have a >> number of different transforms and I am not sure that working with an array >> of codepoints is actually any easier than simply using the underlying UTF-8 >> string of bytes? >> >> My comment about a 'simple' UTF-8 locale for the core strings was intended >> to provide a subset of 'characters' that do not involve these multiple >> codepoint/glyph problems and I include the case wrapping problems with that >> so that the processing of 'metadata' can be kept as clean as possible. So >> the more complex 'locale' support can be kept to only being used in >> 'content' in much the same way that mbstring does work now? >> > > Substring with boundary analysis [1] is even trickier. e.g. Where to add a > new line if words may not be separated by space characters in some languages > like in Japanese or Chinese. > > The "optimal storage format" problem is already addressed by ICU [2]. > Quote: > While most of ICU works with UTF-16 strings and uses data structures > optimized for UTF-16, there are APIs that facilitate working with UTF-8, or > are optimized for UTF-8, or work with Unicode code points (21-bit integer > values) regardless of string encoding. Some data structures are designed to > work equally well with UTF-16 and UTF-8. Exactly. The point is to be efficient by default for most common operations. It seems that UTF-8 will allow that while it is not the fastest way to process Unicode data, UTF-16 is faster. See http://site.icu-project.org/design/collation/v2/perf However this difference is cancelled by a very large of conversions needed by PHP when it interacts with userland data and other libraries, which mainly deal with UTF-8 input and output. Whether UTF-16 usage and conversion to/from UTF-8 is still valid with recent ICU versions has to be confirmed. However (sorry to reply in one mail instead of each single one in this thread), choosing UTF-8 does not mean we will not provide other features provided by UTF-16 ICU APIs. As stated in the unicode idea section (ideas page), it is possible to have both and conversions to UTF-16 can be done when required. Conversions from or to UTF-8 could be done transparently. The key is to define when and how. How it will deal with interned string in this case is also a big question. I have no special preference yet about the existing string functions. I tend to think that we should do both. What could it mean? - provide a unicode string data type (which form it will take has to be defined, for userland, object, pseudo object, etc.) - keep BC with legacy string data, as we have it now - existing functions can then see what they get and behave accordingly About dealing with CP, glyphs or bytes (as of now) are all special cases. But I can't imagine an application doing a string replacement operation using "Östereich" (whether it uses many CPs or not, just an example) and asking the users to have a clue about how "Ö" is internally stored. That will be awkward. Most other languages actually deal with "visible/printable" strings and provide functions to deal with glyphs or CPs. Cheers, -- Pierre @pierrejoye | http://www.libgd.org