unused regression tests
| From: | Roman Neuhauser | Date: | Mon, 15 Apr 2002 11:39:54 +0000 |
| Subject: | unused regression tests | ||
| Groups: | php.qa | ||
| Request: | Send a blank email to php-qa+get-5108@lists.php.net to get a copy of this message | ||
Hi there,
I have two questions regarding the regression testing framwwork. Quite
simple ones.
1) What good is a regression testing framework when the developers
clearly ignore it?
Example: the exit() behavior change breaking tests left everyone
without worrying.
2) What good is a regression testing framework when the tests themselves
are broken?
Example: any .phpt file that leaves the evaluation to quicktester.inc
is broken: or should I expect include("tests/quicktester.inc") work
when WC-root/tests is not in include_path? The php invoked by make
test gets "-c php.ini-dist" on the command line. The commandlines
created by run-tests.php do not include the -c switch, leaving the
door open to potentially bogus failures.
I wonder why even pretend regression tests are used?
Now for something completely related. run-tests.php is IMO needlessly
big. I'll even call it bloated. Someone was clearly enjoying typing.
This makes it unnecesarily hard to read.
Example: the .phpt files are compiled into tmpfile()s, and, when the
expected result differs from the actual one, copied into foo.php,
foo.out and foo.exp. There's even a shutdown callback that should delete
the temporary files, even though it's useless.
run-tests.php should be IMO as simple as possible. The less code it
contains the smaller chance it'll break. If all is broken, the tests
must work.
--
FreeBSD 4.4-STABLE
12:35PM up 1:48, 6 users, load averages: 0.10, 0.05, 0.01