1) What good is a regression testing framework when the developers
clearly ignore it?
Example: the exit() behavior change breaking tests left everyone
without worrying.
I believe that exit() was behaving in a very non-Un*x way, and they
fixed it, and it was bad of the regression tests to rely on the
former (undocumented?) behaviour...
That's about right, but I maintain that ignoring changes in test
results renders the tests useless. Heck, the test I'm talking about
was left broken for a few months.
Perhaps they weren't ignored -- Perhaps they were used as an indicator of what to check out "by hand" and then their results were simply "false positives" in an early stage of a testing process.
As upset as you seem about this, there's really only one solution I can [again] suggest: Fix the code. :-)
2) What good is a regression testing framework when the tests themselves
are broken?
Example: any .phpt file that leaves the evaluation to quicktester.inc
is broken: or should I expect include("tests/quicktester.inc") work
when WC-root/tests is not in include_path? The php invoked by make
test gets "-c php.ini-dist" on the command line. The commandlines
created by run-tests.php do not include the -c switch, leaving the
door open to potentially bogus failures.
Bogus failures, if they are investigated and cleared are annoying,
but not evil.
The tests exist to reflect the state of PHP, right? If they can't do
that...
Look at it this way: if you configure PHP with --disable-session,
the session tests will run anyway, failing inevitably. Would you
really expect anyone paying attention to failed session tests in PHP
with sessions included? I don't think it's very likely.
Look at it this way: If you configure PHP with --disable-session, and then the tests "fail" and you know it's just because you turned the damn thing off, are you going to get your knickers in a twist about it or just go on with your life :-)
I'm just saying that the people are *watching* the test results, and are aware of a test's limitations. They are simply a first-pass filter of what to look at for testing, not the be-all end-all answer.
That said, they could be improved a whole lot, of course.
I wonder why even pretend regression tests are used?
They actually *ARE* being used now -- Or at least I think that's what
is generating all the "Test results" traffic...
This is really nice and all, but it's not enough. Developers need to
run make test just like occasional contributors are told to in
README.TESTING. I know it can't be forced. I'm just very unhappy
about the state of things.
I believe Developers *do* run make test... At least, I have to assume so. It would be even nicer if they would then fix the test if that's what is broken, but you have to convince them that it is important, and I'm afraid you're not exactly going about that in the best way... :-)
Now for something completely related. run-tests.php is IMO needlessly
big. I'll even call it bloated. Someone was clearly enjoying typing.
This makes it unnecesarily hard to read.
Submit a patch to fix it. :-)
Being worked on.
That would rock!
I agree the tests are important to run and evaluate the results.
I suspect you are wrong about how developers use the tests in their process/procedure for vetting code.
I believe everybody wants better reliability in tests and more of them, but not enough developers have focused much on writing/fixing them.
Hopefully your efforts will inspire others to pitch in and improve "make test"
--
Got Music? http://l-i-e.com/artists.htm