Re: Unicode support

On 14/10/2014 21:18, Aleksey Tulinov wrote:

Back to combining characters, i dig the idea of introducing graphemes, but i think French person would write word "noël" using precomposed character. I'm using French keyboard at https://translate.google.com/#fr/. "ë" is Shift + "^", then "e", it produces precomposed U+00EB.

You don't even need to rely on the input method using the combined form, Unicode includes an algorithm for normalisation to this form (where such composites are coded), known as NFC.

If script doesn't have precomposed equivalent, then this grapheme will always be in the same decomposed form and collation will work. Substring search will also work, because needle will be decomposed in the same way as haystack.

No, it won't. You won't get false negatives as long as both strings are normalised to the same form (whether that is NFC or NFD), but you will get false positives. For instance, searching for the substring "e" would not match a combined ë, but it would match an uncombined sequence with e at its base (e.g. with two diacritics). Normalising to NFD (fully de-composed) would at least mean that "e" consistently matched all graphemes with "e" at their base, but is not a lossless operation, so performing it implicitly is probably not a good idea. All of which ignores the questions of length and string reversal, which I think are much more important in this respect.

There are some border-line cases possible, but are they really practical in a scope of Unicode support in a programming language?

As I understand it, the entirety of the Korean writing system is an "edge case" in this respect - it uses 3 code points for each grapheme, and cutting one of those graphemes apart leaves you with gibberish. It's pretty meaningless to say you support Unicode, but only the easy bits. You might as well just tag each string with one of the pages of ISO-8859. -- Rowan Collins [IMSoP]