PHPRunTests output overview - Draft - 2009-06-25

Hello everyone, A few weeks ago, I asked about writing a draft about how to handle all tests results sent with phpruntests. Here is the first version of the draft; Please feel free to annotate any part you want with your own point of view/comments. Overview -------- Today, all tests results are sent on a mailing list, QA-REPORTS, but there is no automatic parsing nor grouping of tests results. Reading it is not so easy (does anyone read them all ?!). The new run-tests bundled with PHP (phpruntests) is currently available at http://cvs.php.net/viewvc.cgi/phpruntests/. This script will be able to output an xml file, making it easy to parse. This xml will contain informations about succeeded tests, failures, etc. This draft is describing the web interface that would show all results sent by users. The objective is to have a better tool for PHP developers to follow bug reports and give much more details and informations than today. Main features wanted -------------------- These features can be seen as "must have" and is not a limitation of what the tool can do. - Agregate results sent - Precision of data based on PHP release (PHP5.2 tests results must not be merged with PHP5.3 results. Same for beta/rc. What about snapshots ?). - Show summary for each test of total failure, total passed and total skipped. Other features wanted --------------------- Additional features that you would like to see : - Cross data with user platform (32bits/64bits, ppc, mac ...) and language (useful for tests on iconv for example). - Aggregate results on 'diff' sent (if a same test is failing for two users but with two different outputs). - Show which tests related to a specific bug are now OK : Highlight where to remove XFAIL, and gives an easy way to create a changelog with bugfixes. - Overview / statistics on : * tests that passed but are now failing (i.e. regression) * tests always skipped (maybe --SKIPIF-- too strict or not enough feedback from a specific platform/extension ?) * ... - Should this tool cross data about code coverage (gcov) or memory leaks (valgrind enabled) ? - backward compatible if we add features (do not loose every results sent so far) Existing tools -------------- Before writing a new tool from scratch, we should first see if existing softwares can do the job. 1/ CDash == + == Aggregation of sent reports == - == Not much details for each test. => To my mind it has very few features, and the time spent on adapting it for PHP is equivalent to the time needed to recode same functionalities from scratch. 2/ Smolder == + == Graphs and overview == - == Programmed in Perl Aggregation of reports does not seem to match PHP-QA needs. Does anyone already use it and know more ? Decision -------- A decision should be made about this project. It is quite huge, but to my mind can be handled easily if the basics are correctly set. Everyone would be able to help by adding features if the structure is well made. a future GSOC project would also be a good way to improve the tool. The actual GSOC may be about implementing core files (if possible). I don't want to go further in this draft. First, we need to agree on: 1/ __IF__ we decide to get this done whether or not 2/ if we use an existing software 3/ if we don't do anything at all. All feedbacks welcome ! Olivier