Re: --XFAIL--

zoe schrieb:

--XFAIL-- This test will fail until bug #123456 is fix

We have had similar discussions at MySQL during the mysqlnd development. QA folks asked to skip tests and/or mark them as TODO within the test as they aimed for zero test failures all the time. Personally I don't like that approach. Its sort of cheating, its hiding the true amount of test failures. Parts of the discussion got solved by not patching run-tests but introducing an environment variable which controls wheter certain tests for experimental/incomplete features will be run or not. By default we do not run those tests. Even if the test is known to fail there is some value in running the test. For example, if you make a change and the test failure - the .diff file - changes, you learn that your change is likely to cause new issues. I'm not sure if this kind of TODO/SKIP etc. (see also Test Anything Protocol - TAP) is really what you want. Instead of skipping tests and changing the tests itself I would prefer having a test results history. run-tests.php should manage the history. And whenever I do a new test run, whenever I make a change, run-tests.php will diff the current and previous test results. The diff tells me what my last change has broke/fixed and the history could be shared/compared among different test systems to define what results one can expect. Ulf