Regression Tests

At 04:26 AM 8/6/2000, Manuel Lemos wrote: Hello Stig, On 06-Aug-00 06:19:07, you wrote: >> Now, a greater challenge:  many existing bugs lead PHP to crash.  How do >> you (plan to) handle crashes in your test suite?  Since you use PHP scripts >> to test features I supposed you have to use something external, like shell >> scripts or PHP scripts running independently of the actual test scripts, to >> detect failures that lead to crashes. >The idea is very simple: a PHP/shell script runs the PHP interpreter >(with a dedicated php.ini file and/or customization directives on the >command line) once for each test.  The stdout of the test is then >compared to the expected output, if they differ, the test fails.  If it >gets harder than this, I fear people won't bother writing tests. I know how regression tests work, but if you run from shell, it will be hard to run them on non-unix platforms.  OTOH hand, if you run from a single PHP script it terminates when a bug leads PHP to crash.  You have to rely on a PHP only solution to a master PHP script call individual test scripts using only the standalone CGI executable.  Besides that there may exist bugs that only show when PHP is run as a server module, so the scripts may have to runnable that way. A possible option lies in using the output buffer and the 'eval' call. A single master script could iterate among a series of test or case scripts, capturing and comparing output/expected output, examining the output buffer and any other variables at the end of each test.  The only disadvantage would be testing when output_buffering is disabled.  Regardless,  I think this would remove the "is or isn't a server module" issue. I have thought if the script were calling other scripts via a URL through the web server you can isolate the scripts from interfering with the master script, unless you hit a really nasty bug which crashes the server. An example would be something like {pseudocode} you have regression_test directory . .. case1 case2 case3 ... in each case directory script.php options.ini output.txt input.txt expected.txt an option file could contain method = http|eval vars = yes|nor [files] outputfile = {some file if needed} inputfile = {some file if needed} expectedoutputfile = {another file} [vars] var1 = {expected value} . . . each case could dump its variables to the output file? ---- for each directory in regression test directory {         read/parse option file (specifying local eval or get through http:// or other method and any state variables)         evaluate option file to determine if test can be done (exit if no)         clear output buffer         eval or get output from script via method in option file         compare output buffer with expected output         compare variables (if any) with expected variable values (generate diff?)         store results in local vars/file to generate report         output something to browser to keep it alive... {progress report}         flush output buffer } output report when done... success/fails ... etc ...