RE: [INTERNALS-WIN] Question about using UTF-8 encoding for server variables used by PHP
| From: | Wade Hilmo | Date: | Tue, 26 Jan 2010 01:03:10 +0000 |
| Subject: | RE: [INTERNALS-WIN] Question about using UTF-8 encoding for server variables used by PHP | ||
| References: | 1 2 3 4 5 6 7 8 9 | Groups: | php.internals.win |
| Request: | Send a blank email to internals-win+get-678@lists.php.net to get a copy of this message | ||
Hi Pierre,
I think that there is a difference of philosophy that is at the root of the issue here. PHP
considers decoding of the URL to be a task for the application. IIS considers decoding of the URL
to be a server function.
The main reason IIS owns decoding is that the IIS core server takes some actions on the decoded URL
before it even calls into application code. The results of these actions sometimes have significant
security ramifications. Historically, there have been a number of exploited vulnerabilities that
were a result of an application decoding the URL in a way that is not identical to the way that IIS
decodes it. Any change to the way that we do this would be perceived as a security risk (and based
on my experience, it would eventually be exploited in some way.)
I am not familiar with the internals of Apache, but I would suspect that it could have similar
problems. In particular, Apache may need to process a .htaccess file using a path derived from the
URL. For example, I have seen applications that filter URLs using the FilesMatch directive. If an
application decodes the URL in a way that does not result in an exact match to the URL that Apache
used, it seems there is opportunity there for a problem to occur.
Finally, getting back to Unicode APIs, the point I was trying to make about SCRIPT_FILENAME is that
unless PHP uses Unicode APIs to open the script file, it will not be possible to support multiple
code pages for content. The problem is that the non-Unicode APIs do not support UTF-8 and require
the path to be expressed in the default system code page. This is not possible if the URL was
encoded in some other code page, even though IIS itself can handle different code pages.
Thanks,
-Wade
-----Original Message-----
From: Pierre Joye [mailto:pierre.php@gmail.com]
Sent: Friday, January 22, 2010 6:07 PM
To: Wade Hilmo
Cc: Johannes Schlüter; Ruslan Yakushev; Stanislav Malyshev; internals-win@lists.php.net
Subject: Re: [INTERNALS-WIN] Question about using UTF-8 encoding for server variables used by PHP
hi Wade,
2010/1/22 Wade Hilmo <wadeh@microsoft.com>:
> I have some concerns about putting the raw URL into the REQUEST_URI variable.
I think there is a confusion about the problem Ruslan is trying to
solve, manage url like: http://mysite.com/2009/10/русские-символ,
which are encoded as described in the RFC. None of the files actually
use UTF-8, index.php is not suddenly named индекс.php, even for
Wordpress :).
http://mysite.com/2009/10/русские-символ
is actually processed by the
rewrite module and will end (example) as:
http://mysite.com/index.php?year=2009&month=10&title=русские-символ
Or whatever else is defined as rewrite rule.
> If applications are using data from that value to open files, there is a potential for security
> vulnerabilities. The problem can occur because IIS looks up configuration information using the
> canonicalized version of the URL for the request. This process of canonicalization includes
> decoding the URL, but also some other things. For example, the kernel mode HTTP parser underneath
> IIS does the work of deciding the code page of the incoming URL and then hands IIS a Unicode version
> of the URL for processing. If an application starts with the raw URL and does its own decoding, it
> could potentially use a different code page form what the HTTP parser used. If the application
> then opens a file, it could be using a path that does not match the IIS configuration for the
> request. Since some of the configuration can affect security, problems can be introduced. The
> most common security bugs that arise from this problem are bypassing authentication or authorization
> rules, or serving script source code back to the client because a clever URL managed to trick the
> application into opening a script file meant for another script handler.
>
> In general, we would be happy to do the right thing for PHP so that support for non-ASCII URLs
> and filenames works well. It looks like we could potentially get there by ensuring that
> SCRIPT_FILENAME is correct.
Exactly, see php-src/sapi/cgi/cgi_main.c, line 1000+, there is a
simple explanation about how php will process the (f)cgi inputs and
the implementation follows this text.
> Unfortunately, I understand that not all of the file I/O in PHP uses Unicode APIs.
Nothing in PHP uses the Unicode APIs (except the IO_REPARSE
implementation I added in 5.3.1, in TSRM) , and that won't be possible
to change any time soon (5.2.x or 5.3.x).
> This is not a problem if the URL is sent using a code page that matches the server's
> system because our default is to populate all of the server variables using the system's
> default code page. We've introduced an option in the FastCGI 1.5 handler that we just
> released to encode any or all of the server variables using UTF-8 encoding. This is effective at
> preserving the URL for the application, but the non-Unicode file APIs in Windows do not work with
> UTF-8. That means that PHP would have to convert the paths to Unicode and use only Unicode APIs to
> access the file system.
Why introduce such thing now at this stage? URL are encoded and can be
easily decode in PHP applications, I do that since more than a decade
without issues. It was always the responsability of the application to
deal with that. However as long as this option is disabled by default,
it won't break anything.
> I would be interested in hearing your thoughts on these issues.
Cheers,
--
Pierre
@pierrejoye | http://blog.thepimp.net | http://www.libgd.org