Re: [RFC] Better benchmarks for PHP

Hi Nuno, On Sat, Feb 14, 2009 at 3:06 PM, Nuno Lopes <nlopess@php.net> wrote: > Hi, > > Currently PHP's benchmarks are very poor. The PHP interpreter is only timed > against bench.php. Sadly, this doesn't make use of the majority of PHP's > features, principally OO, but also includes and variable-variables. We'd > like to help rectify that, and have created an RFC with some initial ideas. > It's available at �� > �~�p�/��thttp://wiki.php.net/rfc/better_benchmarks > > The idea at first is not to do something perfect the first time. Rather, > we'd like to have something quickly, so as not to lose momentum. A good > first step might be porting some javascript benchmarks to PHP. Based on that > experience, we can go about the larger and more difficult task of porting > some web-apps to make representative benchmarks. > > Comments, criticisms and volunteers especially welcome, > I think it is an interesting idea and I can see the effort you put in assembling all of the info, especially the "what currently exists section". Some questions: It seems like you elude (I'm not criticizing, just observing) to the definition of "better benchmarks". You mention parts of the language that aren't tested, the need for "work" (which we should define further), testing of applications (also to be further defined), testing of the stack and deployment difficulty and hardware requirements. Breaking these out: 1. language aspects that aren't tested - I think you did a great job of listing these out and I'm sure others will contribute. Are we including/omitting extensions? 2. What do we mean by work? Complex calculations? Database activity? Network connections? 3. testing of applications - should we standardize on applications to test? By benchmarking an application, isn't the person reviewing the application required to know the details of said application? If it was a generally-accepted Open Source PHP application I feel those numbers would be valid, but a application created just for this purpose isn't that useful IMHO, other than just for the delta. 4. Testing of the stack/What is the deployment/hardware requirement? What should we be shooting for? We personally test the stack at the colo level (obviously multiple machines) for overall performance (rps to error rate), network connections (or timeouts), CPU, mem, disk I/O but we don't scrutinize PHP at that point unless there is an issue produced by those numbers from above (actually, we do a single host test prior and Unit Tests and network response time tests kick in at build time, but I'm getting off-subject). I think the primary emphasis of a PHP benchmark should be one box as it is testing PHP functionality. However, I would like to see a project (maybe it an offshoot of this; something that stands more on its own) that standardizes how the "stack" is tested, or, at least published numbers against some open application so people could see the results. Thanks, Tom