Re: New run-tests code

On Wed, Sep 9, 2009 at 14:55, Pierre Joye<pierre.php@gmail.com> wrote: > On Wed, Sep 9, 2009 at 2:50 PM, Hannes Magnusson<bjori@php.net> wrote: >> On Wed, Sep 9, 2009 at 13:14, Pierre Joye<pierre.php@gmail.com> wrote: >>> hi Derick, >>> >>> >>> On Wed, Sep 9, 2009 at 12:04 PM, Derick Rethans<derick@php.net> wrote: >>> >>>> Yes, I do. We picked the original behavior so that we can make our tests >>>> work in every environment. It provides a better spread of code and >>>> settings in the wild. >>> >>> We did not do it by design. There are tests (less every day) that are >>> still dependent on the system php.ini, but they should be fixed >>> instead (splited if necessary). >>> >>> The concept of unit tests is not to write generic tests but to test >> >> It is. And it isn't. >> The generic tests do not have --INI-- sections. >> The test that are targeted at special ini obscurity have --INI-- sections. > > > There are no obscure ini settings, or almost none left :). allow_call_time_pass_reference, register_long_arrays, session.bug_*, variables_order, engine, xmlrpc_errors, arg_separator.input ... >>> each single unique cases we can figure out, different ini settings are >>> different cases. It also makes testing and debugging much easier, and >> >> Exactly. And there is no way we can duplicate every single test case >> for the 1million ini variations. > > For a given function there is never dozen of different settings. I But there are engouh. http_build_query() depends on arg_separator.input and arg_separator.output. The database _connect() functions depend on bunch of ini settings. All functions depend on engine, html_errors, xmlrpc_errors and all their friends. Some of these settings we simply have to hardcode. Other we don't have to and the tests should therefore work on them all or get chunked into multiple tests with either --INI-- or --SKIPIF-- > still think we should test each case with the various possible ini > settings. And we do. With the help of the users. We are sucky in predicting what kind of setup the users have. Lets help them help us. > > Doing so will not only help us to: > - finally have a set of tests to valid change in some settings > - take decision quicker to see what can be affected if we remove or > add a setting How can you do that without knowing what settings the user runs under? "If there is no test using INI option xyz then we can remove it"? That doesn't tell us anything. However, we can check the test results and see which INI settings the users actually use. You cannot do that if you nuke the user ini testing. -Hannes