Re: What "Unicode support" really means

hi Rowan, On Tue, Mar 4, 2014 at 11:57 PM, Rowan Collins <rowan.collins@gmail.com> wrote: I skipped most of the points in your post, not because they are not relevant (in contrary), but only to put things back in their context. But I will reply to each point later (not in the next 2 weeks tho' :), > In all cases, the aim is to have a set of functions which "do the right > thing" with text strings. For one part, this is what we did in the previous php 6 attempt, and what should be done as well here, be for the size, iterations, etc. > 2) What can we do with text strings? > **IMHO, how we answer these questions is more important to most users of the > language than what the implementation looks like internally.** This is what ICU and other libraries do and ensure. Each string operation, but byte sizes or codepoint APIs. > 3) Input, output, decoding, encoding, and normalisation > > This seems to be the part of the problem that has got most attention so far, > so I'll just list out a few things that need considering: > > - known vs unknown encodings (e.g. an HTTP POST request may specify the > encoding of its body, but a %-encodded URL can represent any string of > bytes) > - automatic vs manual decoding (should $_POST contain byte arrays or decoded > strings?) > - implicit decoding (as used in Perl 5; is this a good idea, or does it just > lead to programmers getting in trouble when their assumptions fail?) > - error handling (if a program states, or PHP assumes, a string is in a > particular encoding, but it's not valid, what should happen?) > - file access (default to binary strings and allow an encoding to be > specified, or provide separate functions for "binary" vs "text" access?) > - file system access (the need to call Win32 APIs with UTF-16 arguments in > Windows builds) That part is a non issue, it will happen before 6 anyway and is already being done for a couple of functions. No matter what we do, conversions will happen in this case as we are very much unlikely choosing UCS-2 as internal encoding :). > - database connectors (how should PDO, mysqli et al negotiate character > encodings with the engine?) > - extensions in general (what facilities do they need?) > > The push to use UTF-8 as the internal representation stems from the > likelihood that most of these conversions will be from/to UTF-8. > Conceptually, these would still be conversions, since error-handling and > type labelling needs to happen *somewhere*, they would just be conversions > which happen to be very efficient under the hood. > > There are almost certainly yet more issues I haven't thought of here. > Somebody said Unicode felt like it was designed by committee; that may be, > but it's also complicated *because it has to be*. If you really want to > support all the world's languages properly, there is a limit to how far you > can simplify the problems. PHP should aim to simplify them as far as > possible, and no farther. Indeed, there are many open questions. But the key failure in our previous attempt was the internals encoding (UTF-16), which causes slowness, endless amount encoding/decoding everywhere, and some other design mistakes internally. My point is not to say that we should not design the userland part of Unicode support carefully but without a clean, functional and efficient internal implementation, everything else is irrelevant at this point. It is also important to keep in mind that no matter what we do in userland, the internals work remain pretty much the same, the only changes is what we change, update, improve from a userland point of view. This is why I try to first solve what really matters at this stage. While your points are just as important, they only come a bit later in the game. Cheers, -- Pierre @pierrejoye | http://www.libgd.org