Re: run-tests.php

Hello Zoe, now i get the issue. It grabbs more than one line! This means run-tests.php has to be fixed. The whole thing is designed under the assumption that %s catches no new lines and hence only one line. marcus Monday, September 10, 2007, 6:03:54 PM, you wrote: > Hi Marcus >> >> why is the line number important here? If you include the exact error >> message, that is including the function name red and green you would still >> get the error. > The problem is that the %s is agressively matching the whole of two > lines of output. Leaving the explicit line numbers in prevents this > because then the second line won't match on the line number. > I don't like the solution I proposed (for all of the reasons that you > list below), but it does prevent the agressive matching of %s (or rather > .+?). > Another possibility is add in extra lines of comments in the output (and > expected output). > In that case the test case would look something like this:--TEST-- > Showing problem with run-tests.php > --FILE-- > <?php > echo "Warning: something wrong in function red at line 10\n"; > echo $Done 1\n"; > echo "Warning: something wrong in function green at line 13\n"; > echo $Done 2\n"; > echo "Write whatever you like, it will be swallowed at line 16\n"; > echo $Done 3\n"; ?>> > --EXPECTF-- > Warning: something wrong in function %s at line %d > Done 1 > Warning: something wrong in function %s at line %d > Done 2 > So - it makes the output messier but it does fail as it should . If > you have a better solution I'd be glad to hear it :-) >> This is actually an issue I see frequently. We have to put as >> much as possible but nothing that could change because of test layout or >> some weird settings. And in fact line numbers can change when a test has to >> be touched for whatever reason (for example adding ini settings using >> ini_set() or because of haveing to define a variable or adding a new method >> to a class or whatever). Thus filename and line numbers were catched be %s >> and %d but we did not catch the ' on line ' or actual test script name. >> >> marcus >> >> Friday, September 7, 2007, 10:38:11 AM, you wrote: >> >> >>> Hi there >>> >> >> >>> I just fell into a small trap with php-runtests.php. This test case >>> shows it: >>> >> >> >>> --TEST-- >>> Testing regex matching in run-tests.phpt >>> --FILE-- >>> <?php >>> echo "Warning: something wrong in function red at line 10\n"; >>> echo "Warning: something wrong in function green at line 13\n"; >>> echo "Write whatever you like, it will be swallowed at line 16\n"; >>> >> ?>> >> >>> --EXPECTF-- >>> Warning: something wrong in function %s at line %d >>> Warning: something wrong in function %s at line %d >>> >> >> >> >>> This is a silly test but it passes. To make it fail properly it's >>> necessary to put the line numbers in the expected output and not use >>> with %d. We've used %d this way in quite a few tests :-( and I tripped >>> over it an an error case test that has lots of similar lines of output. >>> >> >> >>> %s is substituted by .+? in php-runtests.php and I think the regex >>> matching (preg_match) is working correctly. I've found one file system >>> test which is passing (and shouldn't be) because of this - I'll fix it >>> later today. >>> >> >> >>> If I'm correct in thinking that php-runtest.php is working as designed >>> I think it might be a good idea to put something on the QA web pages >>> (I'm happy to write it) to suggest that recommeded practice is to use >>> exact line numbers in the expected output instead of %d. Now that I'm >>> writing this it seems almost too obvious to state - but it didn't seem >>> so obvious earlier in the year. >>> >> >> >>> Comments? >>> >> >> >>> Zoe Slattery >>> IBM >>> >> >> >> >> >> Best regards, >> Marcus >> >> Best regards, Marcus