Re: [RFC] UString

> On 21 Oct 2014, at 21:42, Rowan Collins <rowan.collins@gmail.com> wrote: > > The only case I can see where a default encoding would be sensible would be where source code > itself is in a different encoding, so that u('literal string') works as expected. This is only a good idea if we can somehow make it file-local. Otherwise if one library uses Latin-1 and another uses UTF-8 for some reason, bang! > 2) Clarify relationship to a "byte string" > > Most of the API acts like this is an abstract object representing a bunch of Unicode code > points. As such, I'm not sure what getCodepage() does - a code page (or more properly encoding) > is a property of a stream of bytes, so has no meaning in this context, surely? The internal > implementation could use UTF-8, UTF-16, or some made-up encoding (like Perl6's "NFG" > system) and the user should never need to know (other than to understand performance implications). > > On the other hand, when you *do* want a stream of bytes, the class doesn't seem to have an > explicit way to get one. The (currently undocumented) behaviour is apparently to spit out UTF-8 if > cast to a string, but it would be nice to have an explicit function which could be passed a > parameter in order to serialise to, say, UTF-16, instead. I agree on both these points. ->toBytes or ->encode with an explicit charset parameter would be good. I don’t see the point of getCodepage(). > 3) The Grapheme Question > > This has been raised a few times, so I won't labour the point, just mention my current > thinking. > > Unicode is complicated. Partly, that's because of a series of compromises in its design; > but partly, it's because writing systems are complicated, and Unicode tries harder than most > previous systems to acknowledge that. So, there's a tradeoff to be made between giving users > what they think they need, thus hiding the messy details, and giving users the power to do things > right, in a more complex way. > > There is also a namespace mess if you insist on every function and property having to declare > what level of abstraction it's talking about - e.g. $codePointLength instead of $length. > > An idea I've been toying with is rather than having one class representing the slippery > notion of "a Unicode string", having (at least) two, closely tied, classes: > CodePointString (roughly = UString right now) and GraphemeString (a higher level abstraction tied to > the same internal representation). > > I intend to mock this up as a set of interfaces at some point, but the basic idea is that you > could write this: > > // Get an abstract object from a byte string, probably a GraphemeString, parsing the input as > UTF-8 > $str = u('some text'); > // Perform an operation that explicitly deals in Code Points > $str = $str->asCodePoints()->normalise('NFC'); > // Get information using a higher level of abstraction > $length = $str->asGraphemes()->length; > // Perform a high-level mutation, then convert right back to a concrete string of bytes > echo $str->asGraphemes()->reverse()->asByteString('UTF-16'); > > Calling asGraphemes() on a GraphemeString or asCodePoints() on a CodePointString would be legal > but a no-op, so it would be safe to accept both as input to a function, then switch to whichever > level the task required. > > I'm not sure if this finds a good balance between complexity and user-friendliness, and > would welcome anyone's thoughts. I’d rather have some grapheme-specific functions and some code point functions on the same class. Make array-like indexing with [] be by code points as you may be able to do that in constant time, and because there might be multiple approaches to choosing graphemes. Have ->codepointAt(), but also ->nthGrapheme() or something like it. There’s no need for grapheme versions of all functions, but others would need them. Though your approach has its own merits. -- Andrea Faulds http://ajf.me/