Edit report at https://bugs.php.net/bug.php?id=64948&edit=1
ID: 64948
Updated by: requinix@php.net
Reported by: neclimdul at gmail dot com
Summary: FILTER_VALIDATE_URL does not see urls with
underscores as valid URLs.
-Status: Open
+Status: Feedback
Type: Bug
Package: Filter related
Operating System: Ubuntu
PHP Version: 5.4.15
Block user comment: N
Private report: N
New Comment:
@neclimdul: You're looking at RFC 2396 which was obsoleted by RFC 3986. If you look there
you'll see the grammar still allows it, however it comes with a caveat:
> A registered name intended for lookup in the DNS uses the syntax defined in
> Section 3.5 of [RFC1034] and Section 2.1 of [RFC1123].
As has been said, the situation for underscores is complicated and somewhat contradictory, both for
URIs and for DNS. Some RFCs allow them and others do not. I'd give a primer but we're
talking about more than just a couple RFCs involved in this. Basically, URIs allow underscores in
hostnames by way of the grammar but with the caveat I mentioned above, DNS as a whole does not allow
it for labels according to that grammar, yet some DNS record types do allow them (notably SRV and
TXT).
But remember here we're specifically talking about URLs.
Can anyone link a functioning *website* that uses an underscore in the domain name? I'm
thinking we tie the status of this request to that: if there is one and it works in browsers then we
allow underscores, otherwise not.
Previous Comments:
------------------------------------------------------------------------
[2018-10-15 16:25:59] spam2 at rhsoft dot net
you can have underscores in the URL but not in the hostname/domain part, that's it
------------------------------------------------------------------------
[2018-10-15 16:18:06] neclimdul at gmail dot com
"if I could only have the lifetime a few people" First, I feel your pain. I'm
remembering more and more exactly why I got here and the wasted days of annoyance tracking down why
all of a site was working great when I used a value but the form failed to validate and then
refactoring an entire automation to support stripping underscores. Sure it would have been better if
when I stated I'd immediately had things not work but I wasn't thinking "oh, linux
directories don't comply with the server section of RF2396 and I better strip those
underscores" I was getting things done like most developers and just thought "paths are
paths, lets just pass these around and yeah that's working I've got bigger problems."
We make these things clear so our future selves and other developers aren't loosing that time
and live slightly happier lives and I'm just trying to make someone's life easier because
filter_var is kinda out on its own even if it is "correct" and for good reasons.
After your response I wanted to make sure I was 100% clear on what FILTER_VALIDATE_URL claimed so I
read through the documentation again and the related RFCs. So I was sure there wasn't a nuance
that I missed. PHP's documentation is brief and filled with exceptions:
http://php.net/manual/en/filter.filters.validate.php
"Validates value as URL (according to » http://www.faqs.org/rfcs/rfc2396), optionally with
required components. Beware a valid URL may not specify the HTTP protocol http:// so further validation may be required to determine the URL uses an
expected protocol, e.g. ssh:// or mailto:. Note that the function will only
find ASCII URLs to be valid; internationalized domain names (containing non-ASCII characters) will
fail."
RFC2396, exclude the protocol, and only ASCII. Kinda weird but sure.
In that RFC there are 2 sections describing the authority component in question. The second
(Server-based Naming Authority) requires the hostname conform to the DNS specification. Being strict
to the RFC and ignoring the implementations that allow it we should not allow underscores.
The first section however is "Registry-based Naming Authority" which is _clearly_ not
supported FILTER_VALIDATE_URL.
To use the RFC's definitions, the authority component of the RFC is described as:
```
authority = server | reg_name
reg_name = 1*( unreserved | escaped | "$" | "," |
unreserved = alphanum | mark
mark = "-" | "_" | "." | "!" | "~" |
"*" | "'" | "(" | ")"
```
Right there in mark underscore is explicitly supported and I assume without looking some other
characters that would probably blow up.
So, there's a bug. filter_var doesn't support protocols, non-ascii, _or_ Registry-based
name authorities. Again, maybe its just more documentation but something _is_ wrong. Additionally,
discrepancies with most url parsing implementations and other parts of PHP's URL parsing would
be really great to document as well because that could lead to real life software bugs.
------------------------------------------------------------------------
[2018-10-15 15:22:11] spam2 at rhsoft dot net
it is that simple - there are RFC's covering that the underscore is not allowed and there are
clients which behave completly weird if you insist using them
if i could only have the lifetime a few people of me wasted because they never remember things
lonmger than a few mnoths leading to sit again with a local development URL containing and
underscore and wasting hours of debugging why things don#t work relieable in MSIE and so on
just don't se underscores - it's that simple
------------------------------------------------------------------------
[2018-10-15 15:13:48] neclimdul at gmail dot com
Woooh blast from the past.
> Definitely won't fix? parse_url() handles underscores quite nicely.
Its been so long but it feels like that discrepancy(that parse_url is fine with underscores and
filter_var isn't) was connected to me reporting this. Like, some tool used directory names to
automate subdomains and parse_url, browsers, and everything else had been happy then a stray
filter_var with VALIDATE_URL killed it all. So... not really nicely.
> you MUST NOT use underscores in your DNS names - it's that simple
I don't think it is that simple. Like aharvey in his initial response, this is spread out over
lots of RFCs and muddy implementation so I think we need to be 100% clear and documented in why this
works the way it does.
First I want to make the DNS argument 100% clear. DNS hostnames disallow but DNS allows (and
requires) names with underscores in some cases. This makes it more clear than I could here. http://domainkeys.sourceforge.net/underscore.html
http://ietf.org/rfc/rfc2782.txt
If the argument is that URL authorities should be A & AAAA record hostnames, you probably have a
point and my RFC lawyering is not up to arguing against it but DNS would not be the reason,
URL's would.
Second, the point from the initial bug report is that these URLs can happen in reality. The fact is
that support for them is more in the "it works" category then the "it
doesn't" with this implementation mostly just falling in with IE.
While I appreciate filter_var's adherence to RFC's, this is a case where real world and
RFC's have a disconnect and at the least should be clearly documented to people stumbling into
this because as Luke pointed out, its not even consistent in this language.
------------------------------------------------------------------------
[2018-10-15 14:30:09] spam2 at rhsoft dot net
you MUST NOT use underscores in your DNS names - it's that simple
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=64948
--
Edit this bug report at https://bugs.php.net/bug.php?id=64948&edit=1