RE: [PHP-QA] [benchmarks] Server-side PHP

Hi Mich If you have missed it, I will be the gsoc-student that will be work with the benchmarking suite this summer. I still have some more exams to finish in school this year and I will begin working more seriously after my first exam this coming Tuesday, but start full-time in June. I know what to do the first period of the work, which basically most is just porting benchmark scripts from mostly JS to PHP and write the CLI-app. The second part of the period involves the web applications which I'm not totally clear with and would need some directions so the work can be efficient and useful. It's no hurry but it's good for me to be able to take a take a look and test things before I start the real work. As I've heard and seen, you are THE man regarding this! ;) Basically, any comments that you feel that I might need to hear that is not documented in the RFC, you can tell me so I can do some useful work, especially if it has to do with the CLI-app since that is the firt I'll start with :) Thank you! Alexander Hjalmarsson -----Original Message----- From: michiaki.tatsubori@gmail.com [mailto:michiaki.tatsubori@gmail.com] On Behalf Of Michiaki Tatsubori Sent: den 13 maj 2009 08:43 To: Paul Biggar Cc: php-qa@lists.php.net Subject: Re: [PHP-QA] [benchmarks] Server-side PHP Hi, So, my suggestion is to start from open-sourced applications like RUBBoS or SugarCRM, modify them to avoid heavy use of external resources like DBs as much as possible, and provide workload scenarios based on realistic scenarios but excluding non-PHP loads like SSL and static file retrieval. On Thu, May 7, 2009 at 5:35 PM, Paul Biggar <paul.biggar@gmail.com> wrote: >> PER-REQUEST PROCESSING >> .. >> >> WARMING-UP >> .. > > Are both of these handled by counting requests-per-second after x > amount of time (as well as the steady state)? Yes. First one is about some workload for measuring the cost of define operators and function declaration statements, which are almost impossible to measure with CLI-based benchmarks. Even in the steady state, a PHP application must, at least logically, perform these tasks for each request, differently from Java application servers because of the per-request-live-range nature of PHP. Allowing benchmark users to flexibly determine "x" and forcing them to show the parameters they used would be sufficient and efficient for the second purpose. Regulating the official warming-up period might be too much for our purpose, especially for the php.net runtime, which doesn't need warming-up so much. >> An idea in my mind is to develop "purified" version of realistic Web >> applications, like SPECweb, but provide load generating scenarios that >> never put overwhelming load on Web servers, by omitting static file >> requests. > > That's a nice idea. I dont know how realistic SPECweb is though. > SpecJVM was shown to be very unrealisitic compared to real programs. > It also suffers from being (very) closed source, making it difficult > for many people here to use it. > > I believe you were working on something with SugarCRM? Right. Both of them have the closed source problem and we cannot use them unless we solve political issues to make them open. SPECweb is well designed in terms of application logic and workload scenarios. Their design policy is well-documented. SugarCRM is a real and popular application and their benchmark kit includes realistic workloads. Unfortunately, while SugarCRM is open-sourced, its benchmark kit is delivered in a closed licensing manner. We can ask them to make it open but we cannot wait for it. We may be able to create yet another benchmark kit for SugarCRM by ourselves. Providing a load generator like JMeter scenarios and DB setup script might be ok for this purpose, without changing the SugarCRM source code, as the DB load seems to be very low according to my experience with it. If we would want to avoid the DB setup, we should modify the source code to emulate DB access to let it access pseudo data in memory. > For example, for the RUBBoS benchmark (if it can be coaxed to work) > requires machines for the DB, for PHP and for the simulated clients, > requires them to provide shell accounts for the 'master' benchmark > instance, and measures memory usage as well as requests-per-second. Is > this the sort of setup you had in mind? While I am not familiar with RUBBoS, it should be better to simplify the set-up as much as possible, for example, hopefully, by providing a light-weight backend simulator replacing DB that needs to set up, and a turn-key load-generator package, which requires only a single host to generate enough loads. Maybe we can omit think-time simulation, for example, for this purpose to reduce the number of load-generating client threads. I don't have a concrete solution for such light-weight environmental set-up but we can challenge it with the "best efforts" of the community. :-> Regards, Mich -- PHP Quality Assurance Mailing List <http://www.php.net/> To unsubscribe, visit: http://www.php.net/unsub.php