Re: run-tests re-write

Olivier Doucet wrote:

Hello, here is my feedback (tested against googlecode trunk with PHP 5.2.3) : - \n missing at the end of error strings (about TEST_PHP_EXECUTABLE or no test given in command line for example). Thanks - I will fix that/

- Warning: preg_match(): Unknown modifier 'a' in /****/src/testcase/rtTestDifference.php on line 182 No idea about this one. It's in a section of code that I took unmodified from the PHP53 version of run-tests...but see later

- Many tests failed and we cannot see why / where (no verbose output ?) <= it seems executed with PHP 5.3 everything is working just fine. I just checked the latest development version of 5.2 and it looks OK, so I guess something has changed between that and 5.2.3. BUT - this is an interesting point. Stefan and I want to use some PHP53 features (namespaces and phar) and, because this is a medium to long term project, we both think it should be for PHP53 and onwards. There should be a check in the code to prevent use at PHP52, the php command line looks like this:

/installed/php runtests.php -p /path/to/php mytest.phpt I think the installed executable (/installed/php ) and the executable being tested (/path/to/php) should both be at PHP53.

I also have many ideas about how to improve testing in PHP with run-tests.php, and this is the perfect time to talk about it :) - Reports sent to PHP website / mailing list qa-reports should be more detailed : We can add in the report all tests results (success, skip, failed) : we can then parse those reports to see tests that are 100% successful, 100% failure, and others, compared to version/revision. A web interface can then be easily made to compare all this data: test results per version, per test, ... Possibilities are infinite as soon as we have the data. We don't have to process everything first, but adding a functionality is easy and we can reprocess all reports to have results quickly. It seems it may have been a perfect GSOC project, unfortunately it's too late... So, this is actually part of the reason I started to re-write. About a year ago I got some way into solving this problem, working with the existing script, but in the end I found it was too hard to extend in reasonable (maintainable, readable) way. If you look at the requirements documentation in phpruntests/documentation you will see that the sort of things that you describe are covered. I agree that this is high priority. The code is structured to be able to allow this - look in src/testcase/output and you will see two classes, rtTestOutputWriterList and rtTestOutputWriterXML. Right now I only use the 'list' and even that's a fairly basic implementation designed to be good enough to test with but that's all, the XML one isn't implemented yet. One of the suggestions in the GSOC project is that the student should take on this aspect if the parallelisation work goes well, so if we get the GSOC student there is (I think) a realistic chance that this will be addressed.

Please give your feedback about this (is it a good or a bad idea and why). They are good ideas and I thinking that we need have a wiki page for requirements. I've added one here http://wiki.php.net/qa/runtests and will update it with ideas and suggestions after they have been dicussed/agreed on this list.

Zoe