Re: [RFC] Better benchmarks for PHP

It certainly would be good to port some benchmarks across to php. Being able to add a table of comparative performance of benchmarks across the various programming langauges (Java for example) to would be a good move IMO - better to have real-world figures benchmarking entire programs/algorithms in PHP against the other languages than just showing how feature X compares. they, the benchmark figures and tables of comparison, might even make their way into the Irish PHP Users Group's PHP for Enterprise/Business whitepaper ;-) [ http://short.ie/phpwhitepaper ] k. On Sun, Feb 15, 2009 at 8:08 PM, Paul Biggar <paul.biggar@gmail.com> wrote: > On Sun, Feb 15, 2009 at 7:56 PM, Ólafur Waage <olafurw@gmail.com> wrote: > > "For now, we should focus on getting the language benchmarked" > > > > Ok, lets do a hypothetical and someone is writing a benchmark for a for > loop > > and does. > > > > for($i = 1; $i <= 1000000; $i++) > > { > > echo $i."\n"; > > } > > > No, this is exactly wrong. We are not benchmarking feature X. We are > benchmarking programs written in PHP. We want whole programs, and we > shouldn't consider how the pieces interact. > > By splitting out the features, and testing them one at a time - that > is how we ended up with bench.php. > > > > > > On Sun, Feb 15, 2009 at 7:04 PM, Paul Biggar <paul.biggar@gmail.com> > wrote: > >> > >> Hi Tom, > >> > >> On Sun, Feb 15, 2009 at 6:08 PM, Tom Melendez <tom@supertom.com> wrote: > >> > 1. language aspects that aren't tested - I think you did a great job > >> > of listing these out and I'm sure others will contribute. Are we > >> > including/omitting extensions? > >> > >> For now, we should focus on getting the language benchmarked, which > >> excludes extensions. Naturally, some things, like arrays and strings > >> at the very least, will get pulled in, and that's fine. I think in an > >> ideal world, we'd have multiple implementations of benchmarks, some > >> using extensions, but lets not get ahead of ourselves. > >> > >> > >> > >> > 2. What do we mean by work? Complex calculations? Database > >> > activity? Network connections? > >> > 4. Testing of the stack/What is the deployment/hardware requirement? > >> > What should we be shooting for? We personally test the stack at the > >> > colo level (obviously multiple machines) for overall performance (rps > >> > to error rate), network connections (or timeouts), CPU, mem, disk I/O > >> > but we don't scrutinize PHP at that point unless there is an issue > >> > produced by those numbers from above (actually, we do a single host > >> > test prior and Unit Tests and network response time tests kick in at > >> > build time, but I'm getting off-subject). > >> > >> We want to measure the speed of the "PHP language" -- whatever that > >> is. The rest of the stack is only useful if it is required to help us > >> measure a typical app (see below). So extensions, sure. Network > >> latency, and IO, no. DBs are trickier. > >> > >> > >> > 3. testing of applications - should we standardize on applications to > >> > test? By benchmarking an application, isn't the person reviewing the > >> > application required to know the details of said application? If it > >> > was a generally-accepted Open Source PHP application I feel those > >> > numbers would be valid, but a application created just for this > >> > purpose isn't that useful IMHO, other than just for the delta. > >> > >> We'd like to benchmark 'typical applications'. That's pretty hard > >> to > >> define. The easiest thing would be to go to freshmeat, find the 10 > >> most popular PHP apps, and turn them into benchmarks. But we need > >> "real" data, from "real" users. This will not be simple. This is > >> why I > >> recommended starting by porting the javascript benchmarks. Its > >> straightforward, and wont take long, and its way better than what we > >> have. After than we can look at making perfect benchmarks. > >> > >> > >> > >> > I think the primary emphasis of a PHP benchmark should be one box as > >> > it is testing PHP functionality. However, I would like to see a > >> > project (maybe it an offshoot of this; something that stands more on > >> > its own) that standardizes how the "stack" is tested, or, at least > >> > published numbers against some open application so people could see > >> > the results. > >> > >> Try SPECweb2005, or RUBBOS (I mentioned them both in the RFC). That's > >> exactly what they do. Which makes them less useful for actually > >> measuring the language speed. > >> > >> > >> > >> > >> Paul > >> > >> > >> > >> -- > >> Paul Biggar > >> paul.biggar@gmail.com > >> > >> -- > >> PHP Quality Assurance Mailing List > >> <http://www.php.net/> > >> To unsubscribe, visit: http://www.php.net/unsub.php > >> > > > > > > > > -- > Paul Biggar > paul.biggar@gmail.com > > -- > PHP Quality Assurance Mailing List <http://www.php.net/> > To unsubscribe, visit: http://www.php.net/unsub.php > > -- http://short.ie/savenenaghhospital/