#46845 [Asn]: Test expected to fail, but pass, dont cause an error

ID: 46845 Updated by: zoe@php.net Reported By: hannes dot magnusson at gmail dot com Status: Assigned Bug Type: *General Issues Operating System: Linux PHP Version: 5.3.0alpha3 Assigned To: zoe New Comment: This discussion did come up when we added the XFAIL option and iirc we decided to go with the current behaviour. That said, I'm happy to change it if we can find a better solution. Currently, as you say, we have the risk that someone could change PHP, the test could start passing, the developer forgets to remove the XFAIL section and then some time later the test starts to really fail but because there is an XFAIL section the failure is ignored. I have checked through all the tests in PHP53 and found no passing test with an XFAIL section, so this hasn't been a problem yet. One solution might be to make tests always FAIL until the XFAIL section is removed. This might be annoying and confusing if you were trying to fix a bug. However it would ensure that the developer remembered to remove the XFAIL section. Another would be to introduce another state "XFAILPASS"? for tests which have an XFAIL section but actually pass? Or maybe "WARN"? Do you have a preference for any of these? On balance my view is still that it's a small enough risk that this isn't worth another hack in an already much-hacked script, on the other hand if there is a clear 'right' answer I will fix it. Previous Comments: ------------------------------------------------------------------------ [2008-12-12 12:20:44] hannes dot magnusson at gmail dot com Description: ------------ When test using XFAIL start "working" (i.e. don't fail) run-tests.php doesn't mark them in any special manners. This means that when the fix breaks the test goes back into the "Expected fail" category without anyone ever noticing that something broke. ------------------------------------------------------------------------ -- Edit this bug report at http://bugs.php.net/?id=46845&edit=1