Re: [benchmarks] Time measurement CLI-bench

Hi Alexander, On Fri, May 22, 2009 at 5:25 PM, Alexander Hjalmarsson <hjalle@sgh.se> wrote: > Is it a good or bad idea to let the tests run for a period of time, let's > say running deltablue for 10 seconds (or all in 10 seconds) and count how > many times it can finish the test in that period? Typically, we learn nothing by repeating a benchmark run. The normal unit of a benchmark is how much time it takes to run, in seconds. For a web-apps, its more useful to do requests-per-second, but for the CLI, best to stick to the standard measure. Repeating a test can be useful for a benchmark which "warms up", as it allows you compare the peak speed with the initial speed. PHP has a "warm up" period which includes initializing the VM, parsing the source and generating bytecodes. Eliminating the warm-up would allow you compare the results to the best theoretical time for warm-up (ie 0 seconds). However, you can do this without repetition. > The problem is that I don't really know how much "time" it takes to start > the script and parse the file etc, so the idea could really be terrible, > that why I ask here :) A difficulty of benchmarking is that its hard to know what you want to measure. For some people, parsing is really important, for others, only execution is. Start with measuring whatever is easiest, and we can add finer granularity later if people need something specific. > And another question regarding cvs.. Can I just commit the changes I do and > upload the new files? Is it possible to move or restructure files? I thought > of making a test directory where the tests should be, and it would be stupid > to have two versions of the same file. You should be working from the php-benchmarks directory, and so you can do whatever you like there. I'll review your commits after you do them, so try to separate commits into logically separate units, and give a detailed description in your commit log. If the commit is not ready to be reviewed, mention it in the commit log, and I can review a few combined commits later. Thanks, Paul -- Paul Biggar paul.biggar@gmail.com