Test platform coverage requirements

Hi This is a follow up to a couple of IRC discussions from the last few days. At the moment, when people submit test reports after running 'make test' the reports are sent to the http://news.php.net/php.qa.reports list if they are for a release which is listed here http://qa.php.net/. The reason that we ask people to send in reports is so that we can cover platforms that are hard for people to test on routinely. However, the way that the information is presented at the moment makes it quite hard to get a good idea of what tests are failing. The most basic requirement is to have a table of platform name against PHP release. Clicking on the entries in the table should bring up a list of failed tests. It should then be possible to view the .diff .out and .exp files for the test. We already have this at http://gcov.php.net/viewer.php?version=PHP_5_3&func=tests, but it's for one platform, Linux. To extend this for other platforms I believe we need: (a) To modify run-tests.php to send something more easily parsed than a mail. Probably xml?. (b) Somewhere to send the test results (subdirectory on the gcov machine?). (c) Something to parse process the reports regularly and display them & regular clean up of old report files. (d) Something to create a summary table over all tested platforms. Although we talked about tools to help set this up (CDash, Skoll) getting this basic set of tables in place looks like a fairly straightforward PHP job. Travis already volunteered to add XML generation as a requirement for the new improved run-tests.php, Elizabeth offered Windows test results, there are people who subscribe to this list that already provide results for AIX and OSX. I'm happy to look at (c) and (d) above. Please let me know if you think this is a reasonable way to go - or if I've missed something that's going to make it a lot harder than it seems right now :-) Thanks, Zoe