Re: What "Unicode support" really means

On Thu, Mar 6, 2014 at 9:30 AM, Rowan Collins <rowan.collins@gmail.com> wrote: > On 05/03/2014 07:38, Pierre Joye wrote: >>> >>> **IMHO, how we answer these questions is more important to most users of >>> the >>> language than what the implementation looks like internally.** >> >> This is what ICU and other libraries do and ensure. Each string >> operation, but byte sizes or codepoint APIs. > > > We need to make these decisions *for PHP* though - what, to a PHP > programmer, should strlen($some_unicode_string) do? A good library will give > us multiple options to choose from here. I gave possible solutions in my reply. Maybe it needs some clarification or did you miss it? > >> My point is not to say that we should not design the userland part of >> Unicode support carefully but without a clean, functional and efficient >> internal implementation, everything else is irrelevant at this point. > > > I would tend to disagree: you can't decide on an internal representation, > implementation, etc if you haven't decided what you are trying to achieve. Yes, we can, with the actual numbers and features requirements. > For instance, if you want some common functions to work with grapheme > clusters, then your internal implementation needs to have efficient > processing of those. If we want to support locale-based collations, then > locale should be baked into the design, not added as an afterthought. And so > on. These features are implemented in 3rd party libraries anyway. I do not see how we can even think about implementing such features ourselves without rewriting ICU completely, which makes little sense. > Even the selection of UTF-8 is as much about userland requirements as > internal efficiency: converting to and from an external UTF-8 representation > will be a common operation, so having an internal UTF-8 representation > optimizes for that. Exactly what I am saying since my very first post. And why we now need numbers to see how efficient it is. And then see which operations will require conversions, how and when. The UTF-8 internal storage idea is a proposal, not a definitive solution. This is exactly why I started these discussions. It is nice to see them moving forward by the way :) How all these features will be actually exposed in userland and how existing functions will behave is only a matter about which internals or external libraries APIs can be used. About numbers, I am preparing something that should help to get them, along other things. I will post them as soon as it is somehow usable by someone else than myself. Cheers, -- Pierre @pierrejoye | http://www.libgd.org