Re: utf-8 filenames in phar files.

On Sat, Feb 15, 2014 at 10:53 AM, Yasuo Ohgaki <yohgaki@ohgaki.net> wrote: > On Sat, Feb 15, 2014 at 7:50 PM, Lester Caine <lester@lsces.co.uk> wrote: > > > My previous post did not appear on the list ;) > > > > > > Yasuo Ohgaki wrote: > > > >> A lot of the current confusion does seem to be based around the > >> Windows > >> Wide-API as documented in 'The Problem' section of that document. It > >> would > >> seem that my 'naive' view of simply using UTF-8 strings is thwarted > >> by these > >> problems?-- > >> > >> Unicode is like one name with several encoding. We cannot get away from > >> conversions, normalization especially. > >> > > > > That is why personally I'm just looking at UTF8. Which is enough of a > mine > > field on it's own, but since a large swath of what we are working with > now > > is only UTF8 it does seem to be the right base going forward? > > > > I have problem, too. It seems someone is working on DKIM. > Anyway, there are problems, but UTF-8 is way to go. We just cannot remove > conversions. Normalization has number of issues, including security related > one. > > Regards, > > -- > Yasuo Ohgaki > yohgaki@ohgaki.net > Hi Stas, I just saw that the patch has been merged. There was an objection from Yasuo about normalization in past... What's worse the patch doesn't check UTF-8 correctly. It accepts invalid UTF-8. The correct spec is in https://tools.ietf.org/html/rfc3629#section-4. I have got re2c implementation in jsond scanner : https://github.com/bukka/php-jsond/blob/master/jsond_scanner.re#L122-L134 . As you can see it's very different from the provided impl. I think that accepting ill-formed UTF-8 would be a mistake and as such the patch should be reverted. Thanks Jakub