Re: What "Unicode support" really means

Am 05.03.2014 23:04, schrieb Lester Caine:

Andrea Faulds wrote:

I’m unconvinced that dealing with printed characters instead of codepoints is a good idea generally. It’s necessary when doing certain types of text manipulation, but I don’t think we should do this for substr() and general string manipulation. You can’t always necessarily split a group code points into distinct characters, where a code point joins two characters together visually, for example.

I've flagged up before that there are at least three string lengths. 1 - Number of bytes 2 - Number of codepoints 3 - Number of glyphs But the answer to those figures also depend on other factors :( Initially I thought that for string manipulation switching to UTF32 was a no-brainer, but even that has problems when a code point can actually have a number of different transforms and I am not sure that working with an array of codepoints is actually any easier than simply using the underlying UTF-8 string of bytes? My comment about a 'simple' UTF-8 locale for the core strings was intended to provide a subset of 'characters' that do not involve these multiple codepoint/glyph problems and I include the case wrapping problems with that so that the processing of 'metadata' can be kept as clean as possible. So the more complex 'locale' support can be kept to only being used in 'content' in much the same way that mbstring does work now?

Substring with boundary analysis [1] is even trickier. e.g. Where to add a new line if words may not be separated by space characters in some languages like in Japanese or Chinese. The "optimal storage format" problem is already addressed by ICU [2]. Quote: While most of ICU works with UTF-16 strings and uses data structures optimized for UTF-16, there are APIs that facilitate working with UTF-8, or are optimized for UTF-8, or work with Unicode code points (21-bit integer values) regardless of string encoding. Some data structures are designed to work equally well with UTF-16 and UTF-8. [1] http://www.boost.org/doc/libs/1_55_0/libs/locale/doc/html/boundary_analysys.html [2] http://userguide.icu-project.org/strings/utf-8