Re: [RFC] Better benchmarks for PHP

Hi Tom, On Sun, Feb 15, 2009 at 6:08 PM, Tom Melendez <tom@supertom.com> wrote: > 1. language aspects that aren't tested - I think you did a great job > of listing these out and I'm sure others will contribute. Are we > including/omitting extensions? For now, we should focus on getting the language benchmarked, which excludes extensions. Naturally, some things, like arrays and strings at the very least, will get pulled in, and that's fine. I think in an ideal world, we'd have multiple implementations of benchmarks, some using extensions, but lets not get ahead of ourselves. > 2. What do we mean by work? Complex calculations? Database > activity? Network connections? > 4. Testing of the stack/What is the deployment/hardware requirement? > What should we be shooting for? We personally test the stack at the > colo level (obviously multiple machines) for overall performance (rps > to error rate), network connections (or timeouts), CPU, mem, disk I/O > but we don't scrutinize PHP at that point unless there is an issue > produced by those numbers from above (actually, we do a single host > test prior and Unit Tests and network response time tests kick in at > build time, but I'm getting off-subject). We want to measure the speed of the "PHP language" -- whatever that is. The rest of the stack is only useful if it is required to help us measure a typical app (see below). So extensions, sure. Network latency, and IO, no. DBs are trickier. > 3. testing of applications - should we standardize on applications to > test? By benchmarking an application, isn't the person reviewing the > application required to know the details of said application? If it > was a generally-accepted Open Source PHP application I feel those > numbers would be valid, but a application created just for this > purpose isn't that useful IMHO, other than just for the delta. We'd like to benchmark 'typical applications'. That's pretty hard to define. The easiest thing would be to go to freshmeat, find the 10 most popular PHP apps, and turn them into benchmarks. But we need "real" data, from "real" users. This will not be simple. This is why I recommended starting by porting the javascript benchmarks. Its straightforward, and wont take long, and its way better than what we have. After than we can look at making perfect benchmarks. > I think the primary emphasis of a PHP benchmark should be one box as > it is testing PHP functionality. However, I would like to see a > project (maybe it an offshoot of this; something that stands more on > its own) that standardizes how the "stack" is tested, or, at least > published numbers against some open application so people could see > the results. Try SPECweb2005, or RUBBOS (I mentioned them both in the RFC). That's exactly what they do. Which makes them less useful for actually measuring the language speed. Paul -- Paul Biggar paul.biggar@gmail.com