Bug #64948 [Opn]: FILTER_VALIDATE_URL does not see urls with underscores as valid URLs.

Edit report at https://bugs.php.net/bug.php?id=64948&edit=1 ID: 64948 User updated by: neclimdul at gmail dot com Reported by: neclimdul at gmail dot com Summary: FILTER_VALIDATE_URL does not see urls with underscores as valid URLs. Status: Open Type: Bug Package: Filter related Operating System: Ubuntu PHP Version: 5.4.15 Block user comment: N Private report: N New Comment: "if I could only have the lifetime a few people" First, I feel your pain. I'm remembering more and more exactly why I got here and the wasted days of annoyance tracking down why all of a site was working great when I used a value but the form failed to validate and then refactoring an entire automation to support stripping underscores. Sure it would have been better if when I stated I'd immediately had things not work but I wasn't thinking "oh, linux directories don't comply with the server section of RF2396 and I better strip those underscores" I was getting things done like most developers and just thought "paths are paths, lets just pass these around and yeah that's working I've got bigger problems." We make these things clear so our future selves and other developers aren't loosing that time and live slightly happier lives and I'm just trying to make someone's life easier because filter_var is kinda out on its own even if it is "correct" and for good reasons. After your response I wanted to make sure I was 100% clear on what FILTER_VALIDATE_URL claimed so I read through the documentation again and the related RFCs. So I was sure there wasn't a nuance that I missed. PHP's documentation is brief and filled with exceptions: http://php.net/manual/en/filter.filters.validate.php "Validates value as URL (according to » http://www.faqs.org/rfcs/rfc2396), optionally with required components. Beware a valid URL may not specify the HTTP protocol http:// so further validation may be required to determine the URL uses an expected protocol, e.g. ssh:// or mailto:. Note that the function will only find ASCII URLs to be valid; internationalized domain names (containing non-ASCII characters) will fail." RFC2396, exclude the protocol, and only ASCII. Kinda weird but sure. In that RFC there are 2 sections describing the authority component in question. The second (Server-based Naming Authority) requires the hostname conform to the DNS specification. Being strict to the RFC and ignoring the implementations that allow it we should not allow underscores. The first section however is "Registry-based Naming Authority" which is _clearly_ not supported FILTER_VALIDATE_URL. To use the RFC's definitions, the authority component of the RFC is described as: ``` authority = server | reg_name reg_name = 1*( unreserved | escaped | "$" | "," | unreserved = alphanum | mark mark = "-" | "_" | "." | "!" | "~" | "*" | "'" | "(" | ")" ``` Right there in mark underscore is explicitly supported and I assume without looking some other characters that would probably blow up. So, there's a bug. filter_var doesn't support protocols, non-ascii, _or_ Registry-based name authorities. Again, maybe its just more documentation but something _is_ wrong. Additionally, discrepancies with most url parsing implementations and other parts of PHP's URL parsing would be really great to document as well because that could lead to real life software bugs. Previous Comments: ------------------------------------------------------------------------ [2018-10-15 15:22:11] spam2 at rhsoft dot net it is that simple - there are RFC's covering that the underscore is not allowed and there are clients which behave completly weird if you insist using them if i could only have the lifetime a few people of me wasted because they never remember things lonmger than a few mnoths leading to sit again with a local development URL containing and underscore and wasting hours of debugging why things don#t work relieable in MSIE and so on just don't se underscores - it's that simple ------------------------------------------------------------------------ [2018-10-15 15:13:48] neclimdul at gmail dot com Woooh blast from the past. > Definitely won't fix? parse_url() handles underscores quite nicely. Its been so long but it feels like that discrepancy(that parse_url is fine with underscores and filter_var isn't) was connected to me reporting this. Like, some tool used directory names to automate subdomains and parse_url, browsers, and everything else had been happy then a stray filter_var with VALIDATE_URL killed it all. So... not really nicely. > you MUST NOT use underscores in your DNS names - it's that simple I don't think it is that simple. Like aharvey in his initial response, this is spread out over lots of RFCs and muddy implementation so I think we need to be 100% clear and documented in why this works the way it does. First I want to make the DNS argument 100% clear. DNS hostnames disallow but DNS allows (and requires) names with underscores in some cases. This makes it more clear than I could here. http://domainkeys.sourceforge.net/underscore.html http://ietf.org/rfc/rfc2782.txt If the argument is that URL authorities should be A & AAAA record hostnames, you probably have a point and my RFC lawyering is not up to arguing against it but DNS would not be the reason, URL's would. Second, the point from the initial bug report is that these URLs can happen in reality. The fact is that support for them is more in the "it works" category then the "it doesn't" with this implementation mostly just falling in with IE. While I appreciate filter_var's adherence to RFC's, this is a case where real world and RFC's have a disconnect and at the least should be clearly documented to people stumbling into this because as Luke pointed out, its not even consistent in this language. ------------------------------------------------------------------------ [2018-10-15 14:30:09] spam2 at rhsoft dot net you MUST NOT use underscores in your DNS names - it's that simple ------------------------------------------------------------------------ [2018-10-15 13:47:04] luke at reviews dot co dot uk Definitely won't fix? parse_url() handles underscores quite nicely. ------------------------------------------------------------------------ [2015-08-21 17:14:03] rich at social5 dot com The very Stack Overflow link aharvey@php.net referenced seems to suggest that FILTER_VALIDATE_URL *should* allow underscores, does it not? Regardless, ultimately, FILTER_VALIDATE_URL is used "in the wild" to verify whether or not accessing a URL will load a web resource. There certainly *are* many sites that use underscores in both the main domain as well as the subdomain. Regardless of the "lawyer material" (which one could argue is in favor of "allowing" underscores, anyway), doesn't it make sense to have FILTER_VALIDATE_URL reflect URLs that are actually used? This means that FILTER_VALIDATE_URL is useless for us since it causes our system to reject customers that happen to have an underscore in their URL. Unless this bug is fixed, we'll have to implement a separate solution that will allow the underscore-laden URLs. This doesn't sound like the best solution to me. Wouldn't you agree? ------------------------------------------------------------------------ The remainder of the comments for this report are too long. To view the rest of the comments, please view the bug report online at https://bugs.php.net/bug.php?id=64948 -- Edit this bug report at https://bugs.php.net/bug.php?id=64948&edit=1