Re: run-tests.php

Hi Marcus

why is the line number important here? If you include the exact error message, that is including the function name red and green you would still get the error. The problem is that the %s is agressively matching the whole of two lines of output. Leaving the explicit line numbers in prevents this because then the second line won't match on the line number.

I don't like the solution I proposed (for all of the reasons that you list below), but it does prevent the agressive matching of %s (or rather .+?). Another possibility is add in extra lines of comments in the output (and expected output). In that case the test case would look something like this:--TEST-- Showing problem with run-tests.php --FILE-- <?php

       echo "Warning: something wrong in function red at line 10\n";
       echo $Done 1\n";
       echo "Warning: something wrong in function green at line 13\n";
       echo $Done 2\n";
       echo "Write whatever you like, it will be swallowed  at line 16\n";
       echo $Done 3\n";

?> --EXPECTF-- Warning: something wrong in function %s at line %d Done 1 Warning: something wrong in function %s at line %d Done 2 So - it makes the output messier but it does fail as it should . If you have a better solution I'd be glad to hear it :-)

This is actually an issue I see frequently. We have to put as much as possible but nothing that could change because of test layout or some weird settings. And in fact line numbers can change when a test has to be touched for whatever reason (for example adding ini settings using ini_set() or because of haveing to define a variable or adding a new method to a class or whatever). Thus filename and line numbers were catched be %s and %d but we did not catch the ' on line ' or actual test script name. marcus Friday, September 7, 2007, 10:38:11 AM, you wrote:

Hi there

I just fell into a small trap with php-runtests.php. This test case shows it:

--TEST-- Testing regex matching in run-tests.phpt --FILE-- <?php

      echo "Warning: something wrong in function red at line 10\n";
      echo "Warning: something wrong in function green at line 13\n";
      echo "Write whatever you like, it will be swallowed  at line 16\n";
    

?>>

--EXPECTF-- Warning: something wrong in function %s at line %d Warning: something wrong in function %s at line %d

This is a silly test but it passes. To make it fail properly it's necessary to put the line numbers in the expected output and not use with %d. We've used %d this way in quite a few tests :-( and I tripped over it an an error case test that has lots of similar lines of output.

%s is substituted by .+? in php-runtests.php and I think the regex matching (preg_match) is working correctly. I've found one file system test which is passing (and shouldn't be) because of this - I'll fix it later today.

If I'm correct in thinking that php-runtest.php is working as designed I think it might be a good idea to put something on the QA web pages (I'm happy to write it) to suggest that recommeded practice is to use exact line numbers in the expected output instead of %d. Now that I'm writing this it seems almost too obvious to state - but it didn't seem so obvious earlier in the year.

Comments?

Zoe Slattery IBM

Best regards, Marcus