Re: clarification on the performance of require_once andAPC

Alexey Borzov wrote: > Hi, > > Gregory Beaver wrote: >> Sure. This code: >> s >> file1.php >> <?php >> require_once 'foo.php'; >> ?> >> >> file1.php >> <?php >> $driver = 'foo'; >> require_once $driver . '.php'; >> ?> >> >> is a simplistic example of an unconditional and conditional inclusion of >> the same file. The problem was when classes have cross-file inheritance >> relationships between unconditional and conditional inclusion, sometimes >> the same file was included in different ways on different iterations, >> resulting in erroneous caching of the class definition. This led to >> segmentation faults or worse, random fatal errors due to mis-defined >> classes. If you'd like, I can poke Gopal to find out more detail, but >> it is actually unsolvable as long as we have both top-level and >> conditional require_once. > > So these are deficiencies in a third party product that we want to work > around. OK, what are the *other* third party products that we will treat > the same way? This doesn't make sense to me, could you rephrase the question? >> When I say clever design, I mean doing this kind of check for the driver >> inside MDB2: >> >> <?php >> (assuming we're in namespace PEAR2 and have imported >> PEAR2::MDB2::Exception as Exception) >> if (class_exists('MDB2::mysqli', true)) { >> return true; >> } elseif (file_exists('PEAR2/MDB2/mysqli.php', true)) { >> require 'PEAR2/MDB2/mysqli.php'; >> return true; >> } elseif (file_exists(dirname(__FILE__) . '/MDB2/mysqli.php')) { >> require dirname(__FILE__) . '/MDB2/mysqli.php'; >> return true; >> } else { >> throw new Exception('Cannot locate mysqli driver - have you >> installed PEAR2_MDB2_mysqli package?'); >> } >> ?> >> >> It is my mistake (and a large one) that the language of the RFC does not >> currently allow this kind of design. require should probably be allowed >> as a fallback for loading dependencies if they don't exist, the purpose >> of the new design is to provide more flexibility, not to limit it. > > OK, so we will have this clever check in all packages that should > conditionally include code? Why not create a PEAR2_Loader class then > rather than copy-and-paste? There is no generic way to do this for every package. Mandating an exact way this must be done should not be done unless it really is uniform across all packages. We can provide examples, and if it turns out that 95% or more of packages are doing things the same way, then yes, a loader class would make some sense if and only if it can be shown that it isn't the bottleneck of performance. As I see it, the loader would need to pass in parameters for the classname, exception and package name providing the classname, which would make it pretty slow as a generic case due to the string parsing. Anything that truly solves the problem without restricting other solutions is worth considering in my opinion. > Of course the file shouldn't contain any require_once calls as it will > just contain all the code. Let's enforce the "one package --- one file" > rule: it will both improve performance and ease of use, since even the > most clueless user won't have any problems understanding which file to > include. I realize you're trying to be funny here, but your suggestion is not flexible. "one package --- one file" is inflexible in that you must always include all of the files at once, which is in fact less efficient than including just the part you need. If a package has lots of drivers, which most PEAR packages do, it requires both a lot more memory and more processing time to load the single file. Your counter-proposal provides ease to only one audience, those who want a single-file solution for absolutely everything, and does not help those looking for ultimate performance. >>> And here I'd like to re-ask one of my questions. Currently people can >>> run usage examples and tests right after they install the package. >>> After the proposed changes are implemented will they have to mangle >>> usage examples to make them run? >> No, and in fact if they do, then we need to change the proposal. I >> think one solution might be to relax the replacements restriction for >> documentation and tests files. The new language would read something >> like: >> >> replacements are forbidden in files that are intended to be used in >> production environment (role="php", role="data", role="www") >> but are >> encouraged in documentation examples and in tests in order to >> demonstrate proper usage out-of-the-box. >> >> In other words, by providing two examples and using an install-as tag, >> it becomes possible to provide examples for unzip-and-go and for >> pyrus-installed. [Note: these are spur-of-the-moment examples, I am >> sure better design could be thought of with more thought and >> experimentation by others] >> >> pyrus_example1.php >> <?php >> // if you installed this package using pyrus, this is the example for you >> // we are in docs/PEAR2_MDB2/unzip_example1.php >> set_include_path('@php_dir@'); >> require 'PEAR2/autoload.php'; >> function __autoload($class) >> { >> return PEAR2_autoload($class); >> } >> import PEAR2::MDB2; >> $db = MDB2::connect('mysqli://...'); >> ?> > > This of course nullifies the "PEAR2 should be easily relocatable" claim > made here: http://wiki.pear.php.net/index.php/Pear2_require No, it does not, Alexey. Examples are never used on a production server, and the only code that needs to be easily relocated is the php_dir, data_dir and www_dir code. > Exactly here lies a problem. The distribution model you describe here is > common for *applications*, while most of the stuff in PEAR is currently > *libraries*. I am 99% sure this will be the case with PEAR2 as well. We are also talking about PHP, whereby developers are going to be using the packages. Many PHP developers build their own PHP from source and download stuff. End-users often will install complete PHP applications via apt-get, yum or .msi (think easyphp). Remember also that application developers may want to use PEAR libraries bundled in their applications. All of the steps that make it easier for application developers to distribute their applications with PEAR also make it easier to bundle PEAR libraries inside applications that are NOT installed via the PEAR installer - which is as you point out, the most common use case. It would also make it very easy to upgrade a package that is bundled, a major problem we have with PhpDocumentor (HTML_TreeMenu version -1.3.4 is used currently, and is about 300 years out of date, but upgrading is too painful because of the CVS work). Did it also not occur to you that by making PEAR friendlier to applications that there may be a few more created that are installable by PEAR? I am well aware that PEAR is almost entirely libraries, and the proposals are designed to be used by library authors, of which I am one. I am also a consumer of libraries, and use them in a number of projects that don't involve PEAR at all. My experience with setting up the PEAR environment just to use a single PEAR library has been extremely frustrating in every case. Having the flexibility to download a single library, simply extract it and start to use it would be great. Even better would be if when the application becomes more complex, I could convert this extracted installation into a PEAR installation to update the library. All of this is made possible by the proposal with minimal pain on the part of PEAR developers. Wherever possible, we will be providing some kind of automation of this process, whether it is building an automatic allfiles.php generator into packaging or some other ease-of-development tool like a package skeleton generator that spits out a development module in the manner of ext_skel for PHP. All that I am really asking of you is to have just a tiny bit more vision and to try to work together. > And this even not mentioning the fact that most current *nix users are > more accustomed to some kind of package manager and most Windows users > are accustomed to shiny graphical installers. This is of course irrelevant, as learning any new package management system requires some effort. Were we to instead distribute PEAR packages using yum, apt-get, emerge, windows .msi installer and every other package distribution method (.xpi, anyone?) then your point would be relevant. That is, unless you're volunteering to maintain these installation systems for PEAR packages? >> As I remember it, include_path is an innovation that was introduced into >> PHP in order to support PEAR. Most users use direct inclusion or assume >> that they are simply including files relative to the current directory, >> without knowing that include_path often contains "." > > This "innovation" is available in most other programming languages. So > people who are familiar with programming in general, should understand > what "library path" is about. Then why is it that every day someone comes on to #pear and asks a question that demonstrates they have never heard of it? From today (nickname blanked): [17:35] <***> I'm a little confused, if I use the cli to install a pear package do I still need to download the package to my web_root? [17:45] <***> should including a pear file such as include("DB.php") grab from my pear directory magically? >> Your contention that this proposal is designed to attract stupid people >> is a straw man argument - no one benefits from attracting stupid >> people. However, there are plenty of smart PHP developers who are >> simply not as familiar with include_path because they've never needed to >> know about it. > > My opinion is that solution to include_path problems is not getting rid > of include_path, but educating people about include_path. Here is also > my rant on how stupid package design decisions contribute to these > problems: > http://marc.info/?l=pear-dev&m=118474842223805&w=2 First of all, the proposal is not to get rid of include_path, but instead to make it encouraged but unnecessary to get started. The problem in MDB you point out would have been solved by a better loader design, one that used a forced error (exception) rather than PEAR_Error, and an error message that said exactly which package was missing. BOTH solutions are part of the proposal for PEAR2, and so make your rant irrelevant unless the PEAR2 proposal is not accepted. Now, to rebut the crux of your previous message: > There are essentially 3 issues that need debunking in the proposal > 1) We can get really huge speed improvements with APC ??? Have you read the benchmarks? You linked from a blog entry of Gopal's that talked about one minor change that resulted in a doubling of pages served per second. If we couldn't get huge speed improvements from using APC, who would use it? > 2) include_path is hard (and we can cleanly get rid of it) This is a bogus argument, no one is trying to get rid of include_path, and I have already clearly demonstrated that include_path *IS* hard to understand and to use, you simply ignore it by saying anyone who doesn't is not a programmer. All we are trying to do is make it possible (not recommended - possible) to use PEAR packages as unzip-and-go. Yes, this is a real possibility - PhpDocumentor functions almost as well as an unzip-and-go package as it does installed by PEAR. In fact, it is better when installed by PEAR, as less configuration is needed, but it still works as unzip-and-go if you modify the path inside one file. I have years of experience coding an application to work under both systems, and even more coding the installer and dealing with user's problems grappling with PEAR and include_path. Believe me, it is possible to code applications now using PEAR that work in both settings, but it is inordinately difficult, and these changes would make that far easier. > 3) __autoload() provides some benefits over manual including __autoload() provides a way to rapidly develop an application on a development server, and no performance benefits for the performance-hungry. In other words, your assertion is false, __autoload() can provide the benefit of postponing the process of deciding which files to include() until after you've designed the application, and then it makes it very easy to decide which ones to include by putting a var_dump(get_included_files()) in a shutdown function or by using Gopal's inclued extension. > People who can be educated will read the docs, people who can't be > educated would read neither the docs nor your proposed README files. > You, being American, may also overlook another issue: while the manual > can be (and even *is*) translated, most of the stuff installed alongside > packages will only remain English. The official language of comments and code in PEAR is English, that's why we have translations for the manual. However, this point is irrelevant to your argument. People who can't be educated can't be educated, how does the language affect anything? Right now in PEAR, we have a situation where people can be told in one sentence how to get started. PEAR2 would be no different, but would also provide flexibility for other situations that is not possible for PEAR users. It's just as good as PEAR, only better. >> All that the new proposal mandates is that we stop using top-level >> require_once. The rest is experimental solutions to doing things, and >> all possibilities should be explored until we find one that makes coding >> with PEAR2 packages just as easy as coding with PEAR packages. > > OK, the $100 question: if such a possibility is not found, will you > consider dropping the proposal? Of course. When such a possibility is found, will you be willing to recognize it? >> One obvious example of a process that would be far simpler without the >> problem of top-level require_once munging is the go-pear script. >> Currently, it has to do all kinds of non-obvious weird stuff to get >> things downloaded, and additionally is very difficult to read. If we >> remove all the require_once, go-pear could literally use the PEAR >> installer itself by cramming all of the needed files into a single file, >> and putting the installation script at the top of the file. This would >> even allow us to consider not using a phar archive for PEAR installation >> within PHP, but still supporting a single-file installation model. > > go-pear script is an edge case. I don't think the solution that makes it > easier to do some pretty rare task while making it harder to perform > most other tasks is a good one. Fortunately, the proposal in question makes it easier to perform most common tasks as well as uncommon ones. If you think you can provide a better counter-proposal that solves the problems that have been raised, please do so. Greg