Re: [RFC] Better benchmarks for PHP

"For now, we should focus on getting the language benchmarked" Ok, lets do a hypothetical and someone is writing a benchmark for a for loop and does. for($i = 1; $i <= 1000000; $i++) { echo $i."\n"; } Is this really a benchmark for a for loop or is it too connected to the echo to actually count. I say this because looking at the bench.php file and many of the tests are doing things that they really shouldn't do (like printing out results in a loop). If you agree then this is something that needs to be considered, because a benchmark for feature X should not depend on some other language element. Olafur On Sun, Feb 15, 2009 at 7:04 PM, Paul Biggar <paul.biggar@gmail.com> wrote: > Hi Tom, > > On Sun, Feb 15, 2009 at 6:08 PM, Tom Melendez <tom@supertom.com> wrote: > > 1. language aspects that aren't tested - I think you did a great job > > of listing these out and I'm sure others will contribute. Are we > > including/omitting extensions? > > For now, we should focus on getting the language benchmarked, which > excludes extensions. Naturally, some things, like arrays and strings > at the very least, will get pulled in, and that's fine. I think in an > ideal world, we'd have multiple implementations of benchmarks, some > using extensions, but lets not get ahead of ourselves. > > > > > 2. What do we mean by work? Complex calculations? Database > > activity? Network connections? > > 4. Testing of the stack/What is the deployment/hardware requirement? > > What should we be shooting for? We personally test the stack at the > > colo level (obviously multiple machines) for overall performance (rps > > to error rate), network connections (or timeouts), CPU, mem, disk I/O > > but we don't scrutinize PHP at that point unless there is an issue > > produced by those numbers from above (actually, we do a single host > > test prior and Unit Tests and network response time tests kick in at > > build time, but I'm getting off-subject). > > We want to measure the speed of the "PHP language" -- whatever that > is. The rest of the stack is only useful if it is required to help us > measure a typical app (see below). So extensions, sure. Network > latency, and IO, no. DBs are trickier. > > > > 3. testing of applications - should we standardize on applications to > > test? By benchmarking an application, isn't the person reviewing the > > application required to know the details of said application? If it > > was a generally-accepted Open Source PHP application I feel those > > numbers would be valid, but a application created just for this > > purpose isn't that useful IMHO, other than just for the delta. > > We'd like to benchmark 'typical applications'. That's pretty hard to > define. The easiest thing would be to go to freshmeat, find the 10 > most popular PHP apps, and turn them into benchmarks. But we need > "real" data, from "real" users. This will not be simple. This is why I > recommended starting by porting the javascript benchmarks. Its > straightforward, and wont take long, and its way better than what we > have. After than we can look at making perfect benchmarks. > > > > > I think the primary emphasis of a PHP benchmark should be one box as > > it is testing PHP functionality. However, I would like to see a > > project (maybe it an offshoot of this; something that stands more on > > its own) that standardizes how the "stack" is tested, or, at least > > published numbers against some open application so people could see > > the results. > > Try SPECweb2005, or RUBBOS (I mentioned them both in the RFC). That's > exactly what they do. Which makes them less useful for actually > measuring the language speed. > > > > > Paul > > > > -- > Paul Biggar > paul.biggar@gmail.com > > -- > PHP Quality Assurance Mailing List <http://www.php.net/> > To unsubscribe, visit: http://www.php.net/unsub.php > >