Re: unused regression tests

> Date: Tue, 16 Apr 2002 17:37:05 -0500 > To: php-qa@lists.php.net > From: Richard Lynch <ceo@l-i-e.com> > Subject: Re: [PHP-QA] unused regression tests > > >> >1) What good is a regression testing framework when the developers > >> > clearly ignore it? > >> > Example: the exit() behavior change breaking tests left everyone > >> > without worrying. > >> > >> I believe that exit() was behaving in a very non-Un*x way, and they > >> fixed it, and it was bad of the regression tests to rely on the > >> former (undocumented?) behaviour... > > > > That's about right, but I maintain that ignoring changes in test > > results renders the tests useless. Heck, the test I'm talking about > > was left broken for a few months. > > Perhaps they weren't ignored -- Perhaps they were used as an > indicator of what to check out "by hand" and then their results were > simply "false positives" in an early stage of a testing process. > > As upset as you seem about this, there's really only one solution I > can [again] suggest: Fix the code. :-) > > >> >2) What good is a regression testing framework when the tests themselves > >> > are broken? > >> > Example: any .phpt file that leaves the evaluation to quicktester.inc > >> > is broken: or should I expect include("tests/quicktester.inc") work > >> > when WC-root/tests is not in include_path? The php invoked by make > >> > test gets "-c php.ini-dist" on the command line. The commandlines > >> > created by run-tests.php do not include the -c switch, leaving the > >> > door open to potentially bogus failures. > >> > >> Bogus failures, if they are investigated and cleared are annoying, > >> but not evil. > > > > The tests exist to reflect the state of PHP, right? If they can't do > > that... > > Look at it this way: if you configure PHP with --disable-session, > > the session tests will run anyway, failing inevitably. Would you > > really expect anyone paying attention to failed session tests in PHP > > with sessions included? I don't think it's very likely. > > Look at it this way: If you configure PHP with --disable-session, > and then the tests "fail" and you know it's just because you turned > the damn thing off, are you going to get your knickers in a twist > about it or just go on with your life :-) I said... or meant to say that false positives are bound to cause *real* failures to be missed. > I'm just saying that the people are *watching* the test results, and > are aware of a test's limitations. They are simply a first-pass > filter of what to look at for testing, not the be-all end-all answer. > > That said, they could be improved a whole lot, of course. > > >> >I wonder why even pretend regression tests are used? > >> > >> They actually *ARE* being used now -- Or at least I think that's what > >> is generating all the "Test results" traffic... > > > > This is really nice and all, but it's not enough. Developers need to > > run make test just like occasional contributors are told to in > > README.TESTING. I know it can't be forced. I'm just very unhappy > > about the state of things. > > I believe Developers *do* run make test... At least, I have to > assume so. I would like to believe that too. Reading some commit messages tells me at least some of them don't. > It would be even nicer if they would then fix the test if that's what > is broken, but you have to convince them that it is important, and I'm > afraid you're not exactly going about that in the best way... :-) I know. :) -- FreeBSD 4.4-STABLE 7:52AM up 1 day, 21:04, 10 users, load averages: 0.02, 0.02, 0.00