Re: What "Unicode support" really means

Pierre Joye wrote:

The "optimal storage format" problem is already addressed by ICU [2].

Quote: While most of ICU works with UTF-16 strings and uses data structures optimized for UTF-16, there are APIs that facilitate working with UTF-8, or are optimized for UTF-8, or work with Unicode code points (21-bit integer values) regardless of string encoding. Some data structures are designed to work equally well with UTF-16 and UTF-8. Exactly. The point is to be efficient by default for most common operations. It seems that UTF-8 will allow that while it is not the fastest way to process Unicode data, UTF-16 is faster.

Firebird's first pass at this used the '3 bytes per character' approach, on the basis that storage is cheap, and the record storage process it uses compresses out the 'blank' space. But like the UTF-16 approach it was never totally successful due to the conversion overheads! Byte orientation on UTF-16 being one of the bottlenecks. There are simply two types of string handling that we need to worry about. Simply comparing case sensitive names UTF-8 just works and there is very little internal work to do? It's exactly which path is used to process case comparison which affects internal text strings that is the sticking point? Coming into this 'area' is ordering of strings, which is where a particular locale becomes important. But just what operations does that actually effect? Is it just a process that comes as part of the display conversion or is it more deeply embedded in the 'core'? Personally I already handle this via the database, since I only want the correct subset of data to display, but that is not a generic solution :( The second one wants to carry out more 'character' related processing, which IS currently relegated to mbstring ( or again perhaps the database engine? ) but that does not affect the bulk of program flow? I'm perfectly happy that at this level of processing the switch is to UTF-32 which also seems to be the general direction the ICU is developing? I do not seem to be getting an answer to the one bit I can't get through my own limited delving into the code. Just what 'character handling' operations that need to be codepoint aware are required for the 'program' operations rather than simply handling 'data' strings? -- Lester Caine - G8HFL ----------------------------- Contact - http://lsces.co.uk/wiki/?page=contact L.S.Caine Electronic Services - http://lsces.co.uk EnquirySolve - http://enquirysolve.com/ Model Engineers Digital Workshop - http://medw.co.uk Rainbow Digital Media - http://rainbowdigitalmedia.co.uk