Re: What "Unicode support" really means

Rowan Collins wrote:

On 05/03/2014 08:29, Lester Caine wrote:

Since a large part of the system has already made the move TO UTF-8, running native UTF-8 for many people will never actually need any conversions? My web systems only supply UTF-8 encoded pages these days and everything internally is stored the same, only hitting problems where legacy windows storage is accessed.

So, we don't need to fix anything, then, right? ;) We can't guarantee that the whole world is UTF-8, though, so conversion to other Unicode encoding and "legacy" codesets needs to be supported, and supported well. I think conversion to/from UTF-8 should be treated the same way, even if all it does internally is check the string for invalid encoding and change the type flag: it provides an explicit dividing line between "treat this string as raw bytes with no meaning" and "treat this string as meaningful text, and don't let me mess with the bytes in a way that would corrupt it".

There is currently a mess anyway when someone throws in data that we don't know the encoding. But that is happening less and less? By working UTF-8 internally, one only has the same problem when the source is miss coded?

The first question that still has not been answered is whether there is a general consensus that the whole core should cleanly support UTF-8 or be restricted to a single byte subset? Don't mix 'locale' up with this! We need an acceptable 'global' set of rules which the core works to, for which a 'single byte locale' MAY be the preferred option for everyone?

I'm not really sure what you mean by this. There is no "single-byte subset" of UTF-8, other than it being backwards compatible to 7-bit ASCII. UTF-8 represents a code point with between 1 and 6 bytes.

One of the proposals has been to retain ASCII for all of the 'program' related strings, and this was just a proposal for a 'half way house' solution where the processing is UTF-8, but we work internally with a 'locale' that only worries about case insensitivity for the 'single byte' subset of characters?

I currently see UTF-8 strings but with a 'locale' which only allows a simple 'case insensitive' mapping (no string length changes!). However there IS still an open question on 'case insensitive' and dropping that would remove one of the straight jackets for the UTF-8 side ...

I'm not sure why case sensitivity is any more pressing an issue than the many other complexities of Unicode. Am I missing something?

See other post ... The PROGRAM side of string operations allow case insensitive function and variable names which gets totally screwed if a case conversion also changes the string length! Handling unicode data IN variables is a no brainer. It's using unicode strings to describe the variable that is the sticking point. -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk