Re: Test platform coverage requirements

Nuno Lopes wrote:

Hi, In fact this was supposed to be completed two years ago by my SoC student. Unfortunatelly it wasn't and still remains in my todo list.. My idea was not to use any external software, not because the NIH syndrome, but because we already have the software in place. So my idea is to use the current gcov website's code to display other's machine test results. The results would be stored in the DB in the same way as they are currently stored, but with some machine ID (I was thinking in opening this service for trusted machines only). It seems reasonable. But how do machines get to be trusted? Would you modify run-tests.php so that it only attempts to post if the machine is known? If not how would you deal with input from non-trusted systems?

So, if you want to go this way you have two choices: do it yourself and get moderate help from me until July, This suits me. I have a bunch of other things to do so may not get to it either but I have a little free time at the moment. I read your technotes on gcov but am not that familiar with the code so I have a quite a bit more looking around to do.

or wait for July and pray so that I won't have anything more important to do and actually implement this. OK - it's a back up plan.

If you decide to implement this from scratch with some other program, I would be happy to help integrating it in the gcov server. I think using the existing infrastructure is right. Maybe at a later stage it would be nice to have the sort of pretty pictures that CDash supplies, but right now I think we are missing an important piece of test information so it's better to get basic stuff in place first.

Anyway I think we should have a common interface for all test results. Nuno ----- Original Message ----- From: "zoe" <zoe.slattery@googlemail.com> To: "Travis Swicegood" <development@domain51.com>; <auroraeosrose@gmail.com>; <nlopess@php.net>; <stevseear@php.net>; "Derick Rethans" <derick@php.net>; <php-qa@lists.php.net> Sent: Friday, May 16, 2008 12:10 PM Subject: Test platform coverage requirements

Hi This is a follow up to a couple of IRC discussions from the last few days. At the moment, when people submit test reports after running 'make test' the reports are sent to the http://news.php.net/php.qa.reports list if they are for a release which is listed here http://qa.php.net/. The reason that we ask people to send in reports is so that we can cover platforms that are hard for people to test on routinely. However, the way that the information is presented at the moment makes it quite hard to get a good idea of what tests are failing. The most basic requirement is to have a table of platform name against PHP release. Clicking on the entries in the table should bring up a list of failed tests. It should then be possible to view the .diff .out and .exp files for the test. We already have this at http://gcov.php.net/viewer.php?version=PHP_5_3&func=tests, but it's for one platform, Linux. To extend this for other platforms I believe we need: (a) To modify run-tests.php to send something more easily parsed than a mail. Probably xml?. (b) Somewhere to send the test results (subdirectory on the gcov machine?). (c) Something to parse process the reports regularly and display them & regular clean up of old report files. (d) Something to create a summary table over all tested platforms. Although we talked about tools to help set this up (CDash, Skoll) getting this basic set of tables in place looks like a fairly straightforward PHP job. Travis already volunteered to add XML generation as a requirement for the new improved run-tests.php, Elizabeth offered Windows test results, there are people who subscribe to this list that already provide results for AIX and OSX. I'm happy to look at (c) and (d) above. Please let me know if you think this is a reasonable way to go - or if I've missed something that's going to make it a lot harder than it seems right now :-) Thanks, Zoe