A couple of people have touched on this, but I don't think it's been
directly addressed, so I'll ask it more explicitly: what do these terms
mean?
- UAX31-R1-2
- the standard Default-Ignorable Exclusion Profile
- NFC
In plain terms:
**NFC.** Unicode can write some characters more than one way. An o-umlaut
is either one character, U+00F6, or two: a plain o followed by a combining
diaeresis. Normalization Form C is the form that uses the single character
wherever one exists. Requiring NFC means one spelling per name.
**UAX #31.** The Unicode annex that says which characters a programming
language should allow in identifiers: letters, digits, marks and connecting
punctuation, but not general punctuation, symbols or formatting characters.
"R1-2" only means "we follow it with a stated list of changes" rather than
"we follow it exactly".
**Default-Ignorable Exclusion Profile.** One of those stated changes, and a
standard one that Unicode itself defines rather than something I made up:
drop the characters that are invisible by design -- zero-width joiners,
variation selectors and the like.
what is the summary that goes into the manual and error messages?
"Class name doesn't meet UAX31-R1-2" would be about as meaningful to most
people as the infamous "Unexpected T_PAAMAYIM_NEKUDOTAYIM".
You are right that this is the actual deliverable, and I had not written
it. Attempting it, with the standard's name appearing nowhere:
Identifier contains U+00A0 NO-BREAK SPACE, which is not allowed in a
name
Identifier is not valid UTF-8 (invalid byte 0xA9 at offset 0)
Identifier "gro<U+0308>sse" is not in Unicode normalization form C;
write it as "gr<U+00F6>sse"
The third one bears on your second question.
The other thing that I'm not entirely clear on is how much of this is or
should be about *rejecting* names, and how much about *normalising* them
- bearing in mind we already perform some normalisation in the form of
ASCII case folding.
That is the sharpest thing anyone has said in this thread, and I did not
have it clear in my own head. Sorting the four things I have been bundling
together along that axis:
not valid UTF-8 only rejectable; there is nothing to normalise to
invisible chars only rejectable; removing them would change meaning
not NFC either -- and because the engine knows the composed
spelling, a rejection can print it, which gets most
of the benefit of normalising without the engine
quietly editing your source
ASCII case folding already normalisation, and incomplete:
Stra<U+00DF>e and STRA<U+00DF>E are one class,
Stra<U+1E9E>e is another, Strasse is a third
They also have four different costs, which I can now put numbers to, from
4,863 packages and 520,802 files:
not valid UTF-8 2 identifiers
invisible chars 68, and not all accidental -- math-php spells
variables like <U+27EE>1<U+00A0><U+2212><U+00A0>p<U+27EF><U+02E3>,
with U+00A0 inside the name, on purpose
not NFC 0, across 627,515 files in both corpora
case divergence 19, in case-insensitive positions
Bundling four rules with four cost profiles behind one mechanism was the
mistake. Larry, Claude Pache and you have each pushed on a different corner
of the same thing.
Perhaps we need to step back and define the *problem statement* more
clearly, rather than starting with a *solution* and trying to define its
benefits?
Yes. I started from a mechanism and have been arguing backwards from it all
week. Trying it the other way round:
**PHP's identifier rule is expressed in bytes and says nothing about
characters.** Three things follow. Two identifiers a reader cannot tell
apart may be distinct to the engine. An identifier may be text that is not
well-formed in any encoding. And the engine's own case-insensitive
matching, which is a normalisation, is defined over a 26-letter subset of
what an identifier may contain.
Underneath that sits a question nobody has answered: **are non-ASCII
identifiers a supported feature of PHP?** The manual says they are not, and
explains that they work because of how UTF-8 happens to encode. 1,447 of
them, in 25 of the 5,000 most-installed packages, say otherwise -- Vincenty
geodesy in mjaschen/phpgeo, Latte's U+029F prefix in tracy/tracy, Russian
schema accessors in wsdltophp/packagegenerator, and mathematical formulae
as variable names in markrogoyski/math-php.
I do not think any one of those four items justifies a language change on
its own, and I would rather say so than keep hunting for an argument that
makes it sound bigger. Together they say that PHP's identifier rule was
never designed, only inherited, and that the ecosystem has quietly built on
it anyway. Whether that is worth fixing, and in which direction, is a
question for the list rather than for me.
One observation, and then I will stop reaching for mechanisms. What falls
out of your reject/normalise split is not one feature but three, with three
different audiences: a diagnostic for invisible characters, which needs no
opt-in and is exactly what Claude Pache described wanting; a rule about
well-formedness, which has to reckon with symfony/cache; and a conformance
rule about which characters are permitted at all, which is the only part
that would break math-php 888 times and therefore the only part that
plausibly needs opting into. NFC costs nothing either way and can ride
along with whichever of those happens.
But take the problem statement first. I owe the thread that much before
proposing anything further.
Tooling, raw scanner output and per-identifier CSVs, if anyone wants to
check the numbers rather than take them:
https://github.com/Otzie2023/PHP
Regards,
Luca