Re: [RFC] Better benchmarks for PHP

On Sun, Feb 15, 2009 at 7:56 PM, Ólafur Waage <olafurw@gmail.com> wrote: > "For now, we should focus on getting the language benchmarked" > > Ok, lets do a hypothetical and someone is writing a benchmark for a for loop > and does. > > for($i = 1; $i <= 1000000; $i++) > { > echo $i."\n"; > } No, this is exactly wrong. We are not benchmarking feature X. We are benchmarking programs written in PHP. We want whole programs, and we shouldn't consider how the pieces interact. By splitting out the features, and testing them one at a time - that is how we ended up with bench.php. > On Sun, Feb 15, 2009 at 7:04 PM, Paul Biggar <paul.biggar@gmail.com> wrote: >> >> Hi Tom, >> >> On Sun, Feb 15, 2009 at 6:08 PM, Tom Melendez <tom@supertom.com> wrote: >> > 1. language aspects that aren't tested - I think you did a great job >> > of listing these out and I'm sure others will contribute. Are we >> > including/omitting extensions? >> >> For now, we should focus on getting the language benchmarked, which >> excludes extensions. Naturally, some things, like arrays and strings >> at the very least, will get pulled in, and that's fine. I think in an >> ideal world, we'd have multiple implementations of benchmarks, some >> using extensions, but lets not get ahead of ourselves. >> >> >> >> > 2. What do we mean by work? Complex calculations? Database >> > activity? Network connections? >> > 4. Testing of the stack/What is the deployment/hardware requirement? >> > What should we be shooting for? We personally test the stack at the >> > colo level (obviously multiple machines) for overall performance (rps >> > to error rate), network connections (or timeouts), CPU, mem, disk I/O >> > but we don't scrutinize PHP at that point unless there is an issue >> > produced by those numbers from above (actually, we do a single host >> > test prior and Unit Tests and network response time tests kick in at >> > build time, but I'm getting off-subject). >> >> We want to measure the speed of the "PHP language" -- whatever that >> is. The rest of the stack is only useful if it is required to help us >> measure a typical app (see below). So extensions, sure. Network >> latency, and IO, no. DBs are trickier. >> >> >> > 3. testing of applications - should we standardize on applications to >> > test? By benchmarking an application, isn't the person reviewing the >> > application required to know the details of said application? If it >> > was a generally-accepted Open Source PHP application I feel those >> > numbers would be valid, but a application created just for this >> > purpose isn't that useful IMHO, other than just for the delta. >> >> We'd like to benchmark 'typical applications'. That's pretty hard to >> define. The easiest thing would be to go to freshmeat, find the 10 >> most popular PHP apps, and turn them into benchmarks. But we need >> "real" data, from "real" users. This will not be simple. This is why I >> recommended starting by porting the javascript benchmarks. Its >> straightforward, and wont take long, and its way better than what we >> have. After than we can look at making perfect benchmarks. >> >> >> >> > I think the primary emphasis of a PHP benchmark should be one box as >> > it is testing PHP functionality. However, I would like to see a >> > project (maybe it an offshoot of this; something that stands more on >> > its own) that standardizes how the "stack" is tested, or, at least >> > published numbers against some open application so people could see >> > the results. >> >> Try SPECweb2005, or RUBBOS (I mentioned them both in the RFC). That's >> exactly what they do. Which makes them less useful for actually >> measuring the language speed. >> >> >> >> >> Paul >> >> >> >> -- >> Paul Biggar >> paul.biggar@gmail.com >> >> -- >> PHP Quality Assurance Mailing List >> <http://www.php.net/> >> To unsubscribe, visit: http://www.php.net/unsub.php >> > > -- Paul Biggar paul.biggar@gmail.com