Re: run-tests SKIPIF caching can be problematic for third-party extensions

On Thu, Sep 23, 2021 at 4:10 PM Nikita Popov <nikita.ppv@gmail.com> wrote: > On Wed, Sep 15, 2021 at 7:23 PM Jeremy Mikola <jmikola@gmail.com> wrote: > >> I just discovered that run-tests.php was changed to cache SKIPIF >> evaluation >> since 8.1.0beta3[1]. I believe ext-mongodb ran into the same issue as >> mysqli[2], as we use SKIPIF to check that database contents are clean >> going >> into a test. The present solution in core is to check SKIPIF output for >> "nocache" [3,4] and allow individual tests to opt out of caching; however, >> I'm worried that won't be portable for third-party extensions, where we >> test with run-tests.php for each version of PHP that we support. >> >> Is there still time to consider an alternative solution? If "nocache" >> needs >> to remain in place for mysqli, perhaps 8.1.0's run-tests.php can be >> enhanced to consult an environment variable (for maximum BC and playing >> nice with make test) that disables caching entirely. >> > > I'm not sure I understand the setup you're using. The "nocache" tag that > mysqli uses covers the situation where part of the test setup is done in > the SKIPIF section -- because checking whether the test should be skipped > requires doing the same setup as the actual test. If the test does get > skipped, it's still fine to cache that. > > Now, it sounds like your situation is different from that, and you > actually have SKIPIF sections where first one test should be skipped, but > then a later test with identical SKIPIF shouldn't be skipped. Is that > correct? What changes between the time these two tests run? > Independently of that question (which is about whether "nocache" is sufficient if we ignore cross-version compatibility issue) I do agree that an option to disable the skip cache entirely would make sense. Would https://github.com/php/php-src/pull/7510 work for you? Regards, Nikita