Re: What "Unicode support" really means

On 05/03/2014 08:29, Lester Caine wrote:

Since a large part of the system has already made the move TO UTF-8, running native UTF-8 for many people will never actually need any conversions? My web systems only supply UTF-8 encoded pages these days and everything internally is stored the same, only hitting problems where legacy windows storage is accessed.

So, we don't need to fix anything, then, right? ;) We can't guarantee that the whole world is UTF-8, though, so conversion to other Unicode encoding and "legacy" codesets needs to be supported, and supported well. I think conversion to/from UTF-8 should be treated the same way, even if all it does internally is check the string for invalid encoding and change the type flag: it provides an explicit dividing line between "treat this string as raw bytes with no meaning" and "treat this string as meaningful text, and don't let me mess with the bytes in a way that would corrupt it".

The first question that still has not been answered is whether there is a general consensus that the whole core should cleanly support UTF-8 or be restricted to a single byte subset? Don't mix 'locale' up with this! We need an acceptable 'global' set of rules which the core works to, for which a 'single byte locale' MAY be the preferred option for everyone?

I'm not really sure what you mean by this. There is no "single-byte subset" of UTF-8, other than it being backwards compatible to 7-bit ASCII. UTF-8 represents a code point with between 1 and 6 bytes.

I currently see UTF-8 strings but with a 'locale' which only allows a simple 'case insensitive' mapping (no string length changes!). However there IS still an open question on 'case insensitive' and dropping that would remove one of the straight jackets for the UTF-8 side ...

I'm not sure why case sensitivity is any more pressing an issue than the many other complexities of Unicode. Am I missing something? Regards, -- Rowan Collins [IMSoP]