Re: Test suite and user space (Was: Re: [PHP-DEV] I hope this is the last email about this :))

Melvyn Sopacua wrote:

At 00:27 24-10-2002, Yasuo Ohgaki wrote:

PS: If you would like to write INI independent scripts, I suggest you to use php.ini-recommended at least. You don't/ didn't know phps crashing and make test does not work well with php.ini-recommended, nonetheless, you're insisting there is no problems.

This is a totally different issue. If ini settings affect the test suite, it's a bug in the test suite, period. If you depend on a preset environment, you are testing how things behave in that environment, you are not testing how that specific build for the __end user__ will behave once installed.

Do you realize php.ini used by run-tests.php and phpt differs? Nothing to do with between them.

Test authors, can enforce specific settings they need, in the --INI-- section. If other settings affect the tests behavior, it is a good thing (tm) to know this. It may be totally unexpected and shed light on why certain bugs surface.

I'm writing the same thing _number_ of times.... Do you realize php.ini used by run-tests.php and phpt differs? Nothing to do with between them.

I also removed your statement in README.TESTING, which stated that "if a test fails, the user can see if it's a real bug". There are no maybe's in a test suite. It's a bug or it isn't. Putting in maybe's makes it's purpose confusing and unreliable.

That's fine. When I wrote it, there were more than 30% of failed tests due to improper maintenance of phpt, etc. It was unrealistic to check them all at that time. I guess you don't know this fact.

This is one of the reasons, that during QA periods, some major bugs are not uncovered ('file_exists'). People feel like "oh well, the test is probably bogus", and do not report it. Current work being done on the testsuite is IMO the right direction. Tests for every function under whatever environment and the option to post to QA so that the data can now be collected and analyzed.

It should be. That's why I wrote README.TESTING to make test suite better. If the more people understood tests, the more people help to maintain ;)

If you see that the test suite is not working well, with php.ini-recommended, than please report the tests affected. You don't adjust the road, just because certain cars, have questionable steering - you fix the cars.

Do you realize php.ini used by run-tests.php and phpt differs? Nothing to do with between them. The more I get reply, the more I believe we should specify php.ini-dist for run-tests.php. i.e. People are confused. We do test with phpt, but we shouldn't test run-tests.php. It's just a waste of time. Remember, Derick removed the -c option, yet he does not understand the problem with it, even if I've clearly mentioned _MANY_ times. This incident is enough to illustrate that people cannot write ini dependent script even if one has CVS account. Sometimes simple things became harder to understand... -- Yasuo Ohgaki