Re: unused regression tests

> Date: Mon, 15 Apr 2002 17:23:06 -0500 > To: php-qa@lists.php.net > From: Richard Lynch <ceo@l-i-e.com> > Subject: Re: [PHP-QA] unused regression tests > > >1) What good is a regression testing framework when the developers > > clearly ignore it? > > Example: the exit() behavior change breaking tests left everyone > > without worrying. > > I believe that exit() was behaving in a very non-Un*x way, and they > fixed it, and it was bad of the regression tests to rely on the > former (undocumented?) behaviour... That's about right, but I maintain that ignoring changes in test results renders the tests useless. Heck, the test I'm talking about was left broken for a few months. > >2) What good is a regression testing framework when the tests themselves > > are broken? > > Example: any .phpt file that leaves the evaluation to quicktester.inc > > is broken: or should I expect include("tests/quicktester.inc") work > > when WC-root/tests is not in include_path? The php invoked by make > > test gets "-c php.ini-dist" on the command line. The commandlines > > created by run-tests.php do not include the -c switch, leaving the > > door open to potentially bogus failures. > > Bogus failures, if they are investigated and cleared are annoying, > but not evil. The tests exist to reflect the state of PHP, right? If they can't do that... Look at it this way: if you configure PHP with --disable-session, the session tests will run anyway, failing inevitably. Would you really expect anyone paying attention to failed session tests in PHP with sessions included? I don't think it's very likely. > >I wonder why even pretend regression tests are used? > > They actually *ARE* being used now -- Or at least I think that's what > is generating all the "Test results" traffic... This is really nice and all, but it's not enough. Developers need to run make test just like occasional contributors are told to in README.TESTING. I know it can't be forced. I'm just very unhappy about the state of things. > >Now for something completely related. run-tests.php is IMO needlessly > >big. I'll even call it bloated. Someone was clearly enjoying typing. > >This makes it unnecesarily hard to read. > > Submit a patch to fix it. :-) Being worked on. -- FreeBSD 4.4-STABLE 11:16AM up 1 day, 28 mins, 13 users, load averages: 0.00, 0.01, 0.00