[PATCH] Updated run-test.php

Personally, I am a *very* strong believer in having a set of unit tests that run *every* time you build a new executable. The fact that the Win32 build does not do this is bothersome :). Updated run-tests.php and added to the MSVC workspace. The changes should all work on Unix as well. I don't have CVS commit access, so the updated files are attached. Will include "cvs diff" output and a test log separately. (This message is going to be *very* long... :) FILE run-tests.php - (diff would be much longer) ---------------------------------------------------------------------- <?php /* +----------------------------------------------------------------------+ | PHP Version 4 | +----------------------------------------------------------------------+ | Copyright (c) 1997-2002 The PHP Group | +----------------------------------------------------------------------+ | This source file is subject to version 2.02 of the PHP license, | | that is bundled with this package in the file LICENSE, and is | | available at through the world-wide-web at | | http://www.php.net/license/2_02.txt. | | If you did not receive a copy of the PHP license and are unable to | | obtain it through the world-wide-web, please send a note to | | license@php.net so we can mail you a copy immediately. | +----------------------------------------------------------------------+ | (based on version by: Stig Bakken <ssb@fast.no>) | | (based on the PHP 3 test framework by Rasmus Lerdorf) | +----------------------------------------------------------------------+ */ /* * TODO: * - implement module skipping for PEAR * - do not test PEAR components if base class and/or component class cannot be instanciated */ /* * History * ------- * * 2002-05-07 Preston L. Bannister <preston.bannister@cox.net> * * Rewrote and updated to run on Win32. * * Require exact specification of PHP executable to test (no guessing!). * * Require a specific tests/php.ini (rather than whatever is lying around), * and test that this is indeed what we are using. * * Die if any internal errors encountered in test script. * * Regularized output for simpler post-processing of output. * * Optionally output error lines indicating the failing test source and log * for direct jump with MSVC or Emacs. * * Tested on Windows 2000 with the Cygnus Win32 toolkit installed. */ // Write test context information. echo " TIME " . date('Y-m-d H:i:s') . " - start of test run CWD : " . getcwd() . " PHP : $php PHP_SAPI : " . PHP_SAPI . " PHP_VERSION : " . PHP_VERSION . " PHP_OS : " . PHP_OS . " INI actual : " . get_cfg_var('cfg_file_path') . " INI wanted : " . realpath('tests/php.ini') . " "; set_time_limit(0); ob_implicit_flush(); // Don't ever guess at the PHP executable location. // Require the explicit specification. // Otherwise we could end up testing the wrong file! $php = $_ENV['TEST_PHP_EXECUTABLE']; isset($php) or die("FAIL environment variable TEST_PHP_EXECUTABLE must be set to specify PHP executable!\n"); @is_executable($php) or die("FAIL invalid PHP executable specified by TEST_PHP_EXECUTABLE = " . $php . "\n"); // Make sure we are using the proper php.ini. realpath(get_cfg_var('cfg_file_path')) == realpath("tests/php.ini") or die("FAIL test/php.ini was not used!\n"); // Give a fixed point for tests to pull in include files. putenv("TEST_PHP_ROOT=" . getcwd()); // If parameters given assume they represent selected tests to run. $tests_to_run = array(); if (isset($_SERVER['argc'])) { $argc = $_SERVER['argc']; $argv = $_SERVER['argv']; if (1 < $argc) { for ($i = 1; $i < $argc; $i++) { $test_to_run[$argv[$i]] = 1; } } } // Compile a list of all test files (*.phpt). $test_files = array(); { $directories = array(); $directories[] = getcwd(); for ($n = 0; $n < count($directories); $n++) { $path = $directories[$n]; $module = ''; if (ereg('/ext/([^/]+)/',"$path/",$r)) { $module = $r[1]; } $o = opendir($path) or die("Cannot open directory - $path\n"); while ($name = readdir($o)) { if (is_dir("$path/$name")) { if ('.' == $name) continue; if ('..' == $name) continue; if ('CVS' == $name) continue; $directories[] = "$path/$name"; continue; } // Cleanup any left-over tmp files from last run. ereg('[.]tmp$',$name) and @unlink("$path/$name"); // Otherwise we're only interested in *.phpt files. if (!ereg('[.]phpt$',$name)) continue; //echo "Runnable '" . $name . "' in '" . $path . "'\n"; $testfile = "$path/$name"; $test_files[] = $testfile; $module_of_test[$testfile] = $module; } closedir($o); } } $test_results = array(); // Run only selected tests, if specified. if (count($tests_to_run)) { while (list($name,$runnable) = each($tests_to_run)) { if (!$runnable) continue; $test_results[$name] = run_test($php,$name); } exit; } // We need to know the compiled in modules so we know what to test. $modules_compiled = @get_loaded_extensions(); $modules_to_test = array('' => 1); foreach ($modules_compiled as $module) { //echo "MODULE $module\n"; $modules_to_test[$module] = 1; } // Run all tests (or at least those applicable to this PHP build). sort($test_files); $modules_skipped = array(); $module_current = ''; $path_current = ''; foreach ($test_files as $testfile) { $module = $module_of_test[$testfile]; //echo "MODULE '$module' TEST '$testfile'\n"; if ($module_current != $module) { $module_current = $module; echo " --------------------------------------------------------------------- .... " . ($modules_to_test[$module] ? "testing " : "skipped ") . ($module ? "extension: $module" : "generic PHP") . " "; } // Can we run the test for the given module? if (!$modules_to_test[$module]) { $modules_skipped[$module] += 1; $test_results[$testfile] = 'SKIPPED'; continue; } $path = dirname($testfile); if ($path_current != $path) { $path_current = $path; echo ".... directory $path\n"; } // We've gotten this far - run the test! $test_results[$testfile] = run_test($php,$testfile); } // Summarize results if (0 == count($test_results)) { echo "No tests were run.\n"; return; } $n_total = count($test_results); $sum_results = array(); foreach ($test_results as $v) { $sum_results[$v]++; } $percent_results = array(); while (list($v,$n) = each($sum_results)) { $percent_results[$v] = (100.0 * $n) / $n_total; } echo " ------------------------------------------------------------------- TIME " . date('Y-m-d H:i:s') . " - end of test run TEST RESULT SUMMARY ============================= Number of tests : " . sprintf("%4d",$n_total) . " Tests skipped : " . sprintf("%4d (%2.1f%%)",$sum_results['SKIPPED'],$percent_results['SKIPPED']) . " Tests failed : " . sprintf("%4d (%2.1f%%)",$sum_results['FAILED'],$percent_results['FAILED']) . " Tests passed : " . sprintf("%4d (%2.1f%%)",$sum_results['PASSED'],$percent_results['PASSED']) . " ============================= Skipped " . count($modules_skipped) . " extensions "; // // Functions in the following section are to report the running of one test. // function skip_headers($fp) { // "cli" version of PHP does not output headers if (php_sapi_name() == "cli") { return; } while (!feof($fp)) { if (trim(fgets($fp, 1024)) == "") { break; } } } // // Write the given text to a temporary file, and return the filename. // function save_text($directory,$text) { $tmp = tempnam($directory,'phpt.') or die("Cannot create temporary file!\n"); $fp = @fopen($tmp,'w') or die("Cannot open temporary file!\n"); fwrite($fp,$text); fclose($fp); return $tmp; } // // Write an error in a format recognizable to Emacs or MSVC. // function error_report($testname,$logname) { switch (strtoupper(getenv('TEST_PHP_ERROR_STYLE'))) { case 'MSVC': echo realpath($testname) . "(1) : test that failed.\n"; echo $logname . "(1) : test output.\n"; break; case 'EMACS': echo realpath($testname) . ":1: test that failed.\n"; echo $logname . ":1: test output.\n"; break; } } // // Run an individual test case. // function run_test($php,$file) { // Load the sections of the test file. $section_text = array( 'TEST' => '(unnamed test)', 'SKIPIF' => '', ); $fp = @fopen($file, "r") or die("Cannot open test file: $file\n"); $section = ''; while (!feof($fp)) { $line = fgets($fp); // Match the beginning of a section. if (ereg('^--([A-Z]+)--',$line,$r)) { $section = $r[1]; $section_text[$section] = ''; continue; } // Add to the section text. $section_text[$section] .= $line; } fclose($fp); putenv("PHP_TEST=1"); putenv("REDIRECT_STATUS=1"); putenv("CONTENT_LENGTH="); putenv("QUERY_STRING=".(isset($section_text['GET']) ? $section_text['GET'] : "")); $tmpdir = dirname($file); // Check if test should be skipped. // TBD - note that it would be better (or at least faster) to eval() the SKIPIF section. // This would require adjusting all the SKIPIF sections to omit < ? php ? > and to return() truth. if (trim($section_text['SKIPIF'])) { $tmp = save_text($tmpdir,$section_text['SKIPIF']); $output = $php -c tests -f $tmp; @unlink($tmp); $output = trim($output); //echo "SKIPIF " . $section_text['SKIPIF'] . "\nRETURNS '$output'\n"; if (0 == strcmp('skip',$output)) { return 'SKIPPED'; } } $tmp = save_text($tmpdir,$section_text['FILE']); putenv("PATH_TRANSLATED=$tmp"); putenv("SCRIPT_FILENAME=$tmp"); if (isset($section_text["POST"])) { $tmp_post = save_text($tmpdir,$section_text['POST']); putenv("REQUEST_METHOD=POST"); putenv("CONTENT_TYPE=application/x-www-form-urlencoded"); putenv("CONTENT_LENGTH=".filesize($tmp_post)); $out = $php -c tests -f $tmp 2>&1 < $tmp_post; @unlink($tmp_post); } else { putenv("REQUEST_METHOD=GET"); putenv("CONTENT_TYPE="); putenv("CONTENT_LENGTH="); $out = $php -c tests -f $tmp 2>&1; } @unlink($tmp); // Does the output match what is expected? $tested = trim($section_text['TEST']); $output = trim(preg_replace('/^(..+\n)+\n/','',$out)); $wanted = trim($section_text['EXPECT']); $output = preg_replace('/\r\n/',"\n",$output); $wanted = preg_replace('/\r\n/',"\n",$wanted); $ok = (0 == strcmp($output,$wanted)); if ($ok) { echo "PASS $tested\n"; return 'PASSED'; } // Test failed so we need to report details. echo "FAIL $tested (" . basename($file) . ").\n"; $logname = ereg_replace('\.phpt$','.log',$file); $log = fopen($logname,'w') or die("Cannot create test log - $logname\n"); fwrite($log," ---- EXPECTED OUTPUT $wanted ---- ACTUAL OUTPUT $output ---- FAILED "); fclose($log); error_report($file,$logname); return 'FAILED'; } ?> ---------------------------------------------------------------------- END FILE run-tests.php FILE tests/tests.dsp ---------------------------------------------------------------------- # Microsoft Developer Studio Project File - Name="tests" - Package Owner=<4> # Microsoft Developer Studio Generated Build File, Format Version 6.00 # ** DO NOT EDIT ** # TARGTYPE "Win32 (x86) External Target" 0x0106 CFG=tests - Win32 Debug !MESSAGE This is not a valid makefile. To build this project using NMAKE, !MESSAGE use the Export Makefile command and run !MESSAGE !MESSAGE NMAKE /f "tests.mak". !MESSAGE !MESSAGE You can specify a configuration when running NMAKE !MESSAGE by defining the macro CFG on the command line. For example: !MESSAGE !MESSAGE NMAKE /f "tests.mak" CFG="tests - Win32 Debug" !MESSAGE !MESSAGE Possible choices for configuration are: !MESSAGE !MESSAGE "tests - Win32 Debug" (based on "Win32 (x86) External Target") !MESSAGE # Begin Project # PROP AllowPerConfigDependencies 0 # PROP Scc_ProjName "" # PROP Scc_LocalPath "" # PROP BASE Use_MFC 0 # PROP BASE Use_Debug_Libraries 1 # PROP BASE Output_Dir "." # PROP BASE Intermediate_Dir "." # PROP BASE Cmd_Line "NMAKE /f tests.mak" # PROP BASE Rebuild_Opt "/a" # PROP BASE Target_File "tests.exe" # PROP BASE Target_Dir "" # PROP Use_MFC 0 # PROP Use_Debug_Libraries 1 # PROP Output_Dir "." # PROP Intermediate_Dir "." # PROP Cmd_Line "nmake /nologo /f tests.mak" # PROP Rebuild_Opt "/a" # PROP Target_File "__test_run_complete__" # PROP Bsc_Name "" # PROP Target_Dir "" # Begin Target # Name "tests - Win32 Debug" !IF "$(CFG)" == "tests - Win32 Debug" !ENDIF # Begin Source File SOURCE="..\run-tests.php" # End Source File # Begin Source File SOURCE=.\tests.mak # End Source File # End Target # End Project ---------------------------------------------------------------------- END FILE tests/tests.dsp FILE tests/tests.mak ---------------------------------------------------------------------- # # Win32 Makefile to run the PHP unit tests. # # TEST_PHP_EXECUTABLE # Required - must point to the PHP executable to test. # # TEST_PHP_ERROR_STYLE # Optional - specifies error format to output so IDE can jump to test source and log. # Values: MSVC (Microsoft Visual C++), Emacs # all : run-tests BIN=Debug_TS run-tests : set TEST_PHP_EXECUTABLE=$(BIN)\php-cgi.exe set TEST_PHP_ERROR_STYLE=MSVC cd .. && $(BIN)\php-cgi.exe -c tests -f run-tests.php | tee tests.log ---------------------------------------------------------------------- END FILE tests/tests.mak -- Preston L. Bannister http://members.cox.net/preston.bannister/ pbannister on Yahoo Messenger