Edit report at https://bugs.php.net/bug.php?id=64948&edit=1
ID: 64948
Comment by: spam2 at rhsoft dot net
Reported by: neclimdul at gmail dot com
Summary: FILTER_VALIDATE_URL does not see urls with
underscores as valid URLs.
Status: Feedback
Type: Bug
Package: Filter related
Operating System: Ubuntu
PHP Version: 5.4.15
Block user comment: N
Private report: N
New Comment:
so blame that fool, even wikipedia knows
https://en.wikipedia.org/wiki/Hostname
The Internet standards (Requests for Comments) for protocols mandate that component hostname labels
may contain only the ASCII letters 'a' through 'z' (in a case-insensitive
manner), the digits '0' through '9', and the minus sign ('-'). The
original specification of hostnames in RFC 952, mandated that labels could not start with a digit or
with a minus sign, and must not end with a minus sign. However, a subsequent specification (RFC
1123) permitted hostname labels to start with digits. No other symbols, punctuation characters, or
white space are permitted.
While a hostname may not contain other characters, such as the underscore character (_), other DNS
names may contain the underscore.[4][5] Systems such as DomainKeys and service records use the
underscore as a means to assure that their special character is not confused with hostnames
Previous Comments:
------------------------------------------------------------------------
[2018-10-16 09:29:34] luke at reviews dot co dot uk
As requested, the below URL is what caused me to stumble across this "bug"
https://electrictobacconist_co_uk.secure-cdn.visualsoft.co.uk/images/gamucci-refills-p86-15678_image.jpg
------------------------------------------------------------------------
[2018-10-15 16:31:59] requinix@php.net
@neclimdul: You're looking at RFC 2396 which was obsoleted by RFC 3986. If you look there
you'll see the grammar still allows it, however it comes with a caveat:
> A registered name intended for lookup in the DNS uses the syntax defined in
> Section 3.5 of [RFC1034] and Section 2.1 of [RFC1123].
As has been said, the situation for underscores is complicated and somewhat contradictory, both for
URIs and for DNS. Some RFCs allow them and others do not. I'd give a primer but we're
talking about more than just a couple RFCs involved in this. Basically, URIs allow underscores in
hostnames by way of the grammar but with the caveat I mentioned above, DNS as a whole does not allow
it for labels according to that grammar, yet some DNS record types do allow them (notably SRV and
TXT).
But remember here we're specifically talking about URLs.
Can anyone link a functioning *website* that uses an underscore in the domain name? I'm
thinking we tie the status of this request to that: if there is one and it works in browsers then we
allow underscores, otherwise not.
------------------------------------------------------------------------
[2018-10-15 16:25:59] spam2 at rhsoft dot net
you can have underscores in the URL but not in the hostname/domain part, that's it
------------------------------------------------------------------------
[2018-10-15 16:18:06] neclimdul at gmail dot com
"if I could only have the lifetime a few people" First, I feel your pain. I'm
remembering more and more exactly why I got here and the wasted days of annoyance tracking down why
all of a site was working great when I used a value but the form failed to validate and then
refactoring an entire automation to support stripping underscores. Sure it would have been better if
when I stated I'd immediately had things not work but I wasn't thinking "oh, linux
directories don't comply with the server section of RF2396 and I better strip those
underscores" I was getting things done like most developers and just thought "paths are
paths, lets just pass these around and yeah that's working I've got bigger problems."
We make these things clear so our future selves and other developers aren't loosing that time
and live slightly happier lives and I'm just trying to make someone's life easier because
filter_var is kinda out on its own even if it is "correct" and for good reasons.
After your response I wanted to make sure I was 100% clear on what FILTER_VALIDATE_URL claimed so I
read through the documentation again and the related RFCs. So I was sure there wasn't a nuance
that I missed. PHP's documentation is brief and filled with exceptions:
http://php.net/manual/en/filter.filters.validate.php
"Validates value as URL (according to » http://www.faqs.org/rfcs/rfc2396), optionally with
required components. Beware a valid URL may not specify the HTTP protocol http:// so further validation may be required to determine the URL uses an
expected protocol, e.g. ssh:// or mailto:. Note that the function will only
find ASCII URLs to be valid; internationalized domain names (containing non-ASCII characters) will
fail."
RFC2396, exclude the protocol, and only ASCII. Kinda weird but sure.
In that RFC there are 2 sections describing the authority component in question. The second
(Server-based Naming Authority) requires the hostname conform to the DNS specification. Being strict
to the RFC and ignoring the implementations that allow it we should not allow underscores.
The first section however is "Registry-based Naming Authority" which is _clearly_ not
supported FILTER_VALIDATE_URL.
To use the RFC's definitions, the authority component of the RFC is described as:
```
authority = server | reg_name
reg_name = 1*( unreserved | escaped | "$" | "," |
unreserved = alphanum | mark
mark = "-" | "_" | "." | "!" | "~" |
"*" | "'" | "(" | ")"
```
Right there in mark underscore is explicitly supported and I assume without looking some other
characters that would probably blow up.
So, there's a bug. filter_var doesn't support protocols, non-ascii, _or_ Registry-based
name authorities. Again, maybe its just more documentation but something _is_ wrong. Additionally,
discrepancies with most url parsing implementations and other parts of PHP's URL parsing would
be really great to document as well because that could lead to real life software bugs.
------------------------------------------------------------------------
[2018-10-15 15:22:11] spam2 at rhsoft dot net
it is that simple - there are RFC's covering that the underscore is not allowed and there are
clients which behave completly weird if you insist using them
if i could only have the lifetime a few people of me wasted because they never remember things
lonmger than a few mnoths leading to sit again with a local development URL containing and
underscore and wasting hours of debugging why things don#t work relieable in MSIE and so on
just don't se underscores - it's that simple
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=64948
--
Edit this bug report at https://bugs.php.net/bug.php?id=64948&edit=1