Re: [Discussion] Should All S tring Functions Become Multi-Byte Safe?

On 16 August 2024 23:11:18 BST, Nick Lockheart <lists@ageofdream.com> wrote: >I used the rather broad title "Should All String Functions Become >Multi-Byte Safe" because there are many smaller related topics, but my >intention was to discuss multi-byte in general I think it was probably not the best choice, because it seems like what you're specifically interested in is mostly not about existing functions, and not particularly about encodings being more than one byte wide. For instance, even good old 7-bit-per-character ASCII contains control characters you might want help sanitising out; and plenty of 8-bit-per-character encodings include more than one script, even more than one writing direction (e.g. ISO 8859-8 Latin/Hebrew). But, the specific topic of safe input handling is definitely an interesting one. And focussing on Unicode, rather than every possible encoding (multibyte or not) makes sense in modern usage. >There's a lot of potential pitfalls for dealing with Unicode input, and >there are some best practices per the Unicode Consortium It's worth looking into whether the ICU library has explicit functions to help with those recommendations (if you can navigate its slightly patchy documentation). Since most of ext/intl is just a thin wrapper on that library, that could make our lives a lot easier. >For example, there should be a function that removes unassigned code >points. > >There should also be a function that removes "scripts" (as defined by >Unicode). > >We should have an easy way to remove private use code points (unless >you're running a Star Trek fan site and really do need Klingon). These all seem like good ideas. I think you can do at least some of it with regular expressions, but dedicated functions have potential to be both easier to use and more efficient. >And the default replacement character for mb_scrub >shouldn't be ?. This is trickier, and where mixing the terms "multibyte" and "Unicode" actually matters. The mbstring extension supports a number of different text encodings, most of which don't have a dedicated replacement character to use. It also has the ability to set the default in global state with mb_substitute_character() so it's not immediately obvious how a different default could be applied based on the specified encoding. (I'm not a fan of that API design, but it's what we've got!) >Each of these and other ideas could be part of an RFC, or we could >brainstorm a Unicode built-in class that handles lots of the common use >cases. I don't think a single class that tries to "do Unicode" makes sense; it would be like having a "maths class" that contains methods for anything dealing with numbers. In fact, I think the group of functions you're suggesting are a great illustration of what I was saying in my last message to Rob: they make perfect sense as standalone features, and don't need any grand plan to "have Unicode in core" before we proceed with them. Regards, Rowan Tommins [IMSoP]