Edit report at https://bugs.php.net/bug.php?id=36398&edit=1
ID: 36398
Updated by: cmb@php.net
Reported by: nickj-phpbugs at nickj dot org
Summary: "make test" should be fully scriptable [Do not
prompt for email address]
-Status: Open
+Status: Closed
Type: Feature/Change Request
Package: Testing related
Operating System: Linux
PHP Version: 5.1.2
-Assigned To:
+Assigned To: cmb
Block user comment: N
Private report: N
New Comment:
Thanks for the patch, and sorry that it has been missed, but â¦
> Though, I don't know if we really want php-qa list to get filled
> with results of the tests executed automatically every X minutes.
No, we really don't want that. Instead it would be more helpful,
if individual test fails would be investigated by the testers, and
reported as separate tickets, ideally accompanied by a respective
pull request.
Previous Comments:
------------------------------------------------------------------------
[2008-08-07 06:58:48] nickj-phpbugs at nickj dot org
Yes, it would be nice if after calling for a patch, if the submitted patch could be either accepted
or rejected, rather than ignored for 2.5 years ;-)
I gather run-tests.php may be rewritten sometime (I think I saw it somewhere as a possible Google
SoC project), and if this happens including functionality to send the results using an email address
specified on the command line could be nice to include. Presumably then someone (or some bit of
software) can collate the output and help to pinpoint consistent problems, or when things break on
particular platforms, and give a summary of the health of PHP on different platforms, and so forth.
------------------------------------------------------------------------
[2008-08-05 12:21:28] Bjorn dot Wiberg at its dot uu dot se
Shouldn't the patch be incorporated into run-tests.php? As it has been around for a long time.
------------------------------------------------------------------------
[2006-02-17 06:53:53] nickj-phpbugs at nickj dot org
No worries! There is a patch now available for this at:
http://www.files.nickj.org/php/run-tests-patch.txt
Also, the run-test.php script is not currently E_STRICT clean. I have included in the patch some
simple changes I had to make to get it to run with E_STRICT enabled.
Note that there are 3 remaining things I did not know how to fix for clean E_STRICT output, so even
with this patch run-tests.php is still not totally E_STRICT clean. The three remaining things were:
1) Was this error:
Error! type: 8; File: /root/php-5.1-dev/php5.1-200602150330/run-tests.php; Line: 93; Message:
ob_end_clean(): failed to delete buffer. No buffer to delete.
... which was generated by this line under E_STRICT:
while(@ob_end_clean());
2) Was the use of date() in this line:
$output_file = $CUR_DIR . '/php_test_results_' . date('Ymd_Hi') .
'.txt';
Which gives this error under E_STRICT:
Error! type: 2048; File: /root/php-5.1-dev/php5.1-200602150330/run-tests.php; Line: 246; Message:
date(): It is not safe to rely on the system's timezone settings. Please use the date.timezone
setting, the TZ environment variable or the date_default_timezone_set() function. In case you used
any of those methods and you are still getting this warning, you most likely misspelled the timezone
identifier. We selected 'Australia/Melbourne' for 'EST/11.0/DST' instead
Not sure how to say "just use the system default" nowadays, given that we probably
shouldn't hard-code a timezone, as users from anywhere could be running this test script.
3) Was the "select_stream" call in the "system_with_timeout" function, which
generates this error under E_STRICT:
Error! type: 2; File: /root/php-5.1-dev/php5.1-200602150330/run-tests.php; Line: 893; Message:
stream_select(): 230 is not a valid stream resource
------------------------------------------------------------------------
[2006-02-16 07:35:06] tony2001@php.net
Thanks, I've just fixed the typo.
As for the request - I'm pretty sure you can add it to run-tests.php and send a patch for us to
review.
Though, I don't know if we really want php-qa list to get filled with results of the tests
executed automatically every X minutes.
------------------------------------------------------------------------
[2006-02-16 06:19:59] nickj-phpbugs at nickj dot org
Thank you - that sort of helps, and I realise now that I should have phrased my request better (so I
have reopened, and rephrased it to hopefully be clearer).
What I mostly wanted was to run the tests, and then email the results off, without prompting.
I can enable/disable the prompt now with:
TEST_PHP_ARGS="-q" make test
And I can see the available options with:
TEST_PHP_EXECUTABLE="/root/tmp/php-5.1-dev/php5.1-200602150330/sapi/cli/php" sapi/cli/php
run-tests.php --help
However, unless I am mistaken, I can't see any option to specify the email address to use, such
as for example: TEST_PHP_ARGS="--email-results --use-email=my@email.com" make test
My underlying assumption here is that you folks want and use the output of "make test" in
some way. If that's not the case, then of course, don't make it an option. However, if you
are using this, then wouldn't it be good to be able to script it so that it could automatically
email off the results of make test, without having to prompt the user? Then you could (for example)
have the build farm automatically email the qa.php.net list with the results of "make
test" after every build of every snapshot. Maybe you already do this in some other way, but if
not, it seems like it could be useful to me.
You could also update http://qa.php.net/running-tests.php with this
information, so that people could "set and forget" to run the tests and email the results,
without prompting them.
P.s. there is a very small typo in the help for "make test". It says "-q Quite, no
user interaction"; but you want it to say "Quiet", not "Quite".
------------------------------------------------------------------------
The remainder of the comments for this report are too long. To view
the rest of the comments, please view the bug report online at
https://bugs.php.net/bug.php?id=36398
--
Edit this bug report at https://bugs.php.net/bug.php?id=36398&edit=1