Re: Concurrency support for run-tests

When building PFTT, the PHP Full Test Tool, one goal was maximizing speed so that we would be more likely to frequently Fully test PHP (Full-coverage of many scenarios), and to do that I built it to run tests concurrently... Concurrent/parallel execution can cause breaks that otherwise work with normal serial execution, so definitely don't enable concurrency in run-tests by default. run-tests.php's two main benefits are that it supports running the huge library of 15k+pecl PHPTs (compatiblity) and its a lightweight tool so that anyone who just built PHP can run make test to test their build (whereas PFTT Is a heavyweight tool). In making/merging major changes (like concurrency) the most important thing is to not break that compatibility and dependability: Your patch will fail without the /proc filesystem(mac osx?). On Windows, you can check the %NUMBER_OF_PROCESSORS% environment variable. For unices that don't have /proc there may not be a single simple way to check. For better performance, PFTT runs a pool of 3 to 4 processes per CPU, when the processes finish quickly, it runs fewer and when they slow down (due to pending IO ops), it runs more processes. See: http://git.php.net/?p=pftt2.git;a=blob;f=src/com/mostc/pftt/scenario/CLIScenario.java;h=30c81aabcb56698eebe4884d915298c92332b50d;hb=refs/heads/master#l61 and for other SAPI scenarios like Apache it runs 2 or 3 processes, see: http://git.php.net/?p=pftt2.git;a=blob;f=src/com/mostc/pftt/scenario/ProductionWebServerScenario.java;h=d509bddda2b3756f05321ddc273600aeb38372be;hb=refs/heads/master For more speed, rather than grouping all, fe open_basedir tests into a single Concurrency Group its better to have them all operate in separate temporary directories and database tables (then you could run 32 or 64 openbase dir tests at once). The ext/standard/tests/file and ext/mysql* PHPTs take the longest to run and mostly do IO operations so they benefit the most from maximizing concurrency. For databases, PFTT will create a new database (and for phpunit tests populate the database) for each test runner thread, See: http://git.php.net/?p=pftt2.git;a=blob;f=src/com/mostc/pftt/scenario/MySQLScenario.java;h=f768f58c4494cb2705a571ece22d533bfdc15192;hb=refs/heads/master#l164 On Wed, Sep 10, 2014 at 3:36 AM, Michael Wallner <mike@php.net> wrote: > On 09/07/14 22:00, Stas Malyshev wrote: > > Hi! > > > >> Having thought a bit about the subject, the heuristic I came up with > was to > >> 1) do not run tests from the same extension in parallel, and 2) do not > run > >> tests from certain extensions in parallel (e.g., mysql and pdo_mysql). > > > > How about making it a whitelist? For tests/extensions we know it's easy > > to paralellize (like ones not having external dependencies/servers/etc) > > simple script could whitelist a lot of them. For those more tricky, we > > could whitelist them gradually. > > > > So how about adding the concurrency switch, but leaving it disabled by > default? > > -- > Regards, > Mike > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php > >