Extracts from the 'We Want Quality' thread

Hello All, The following is a list of ideas presented by various posters to yesterday's "We Want Quality" thread on the DEV list. I have summarized many of the entries. Please let me know if there is anyone or anything that I have missed. This list should provide us with a baseline to start working on what the QA group will do and how they will do it. ***Notes*** I have not included any of the discussion that focused on code management. If you would like to receive a digest of the 'We Want Quality' thread please send mail to: php-dev-thread.22824@lists.php.net ============================================================ (Sasha) Started the discussion :) (Jason ) Suggested forming a QA Team (Rick Rodgers) 1) Pick the most "important" target platforms Delegate a contact person for this platform Announce these contacts via available channels 2) The contact person should be committed to fully installing the new release on her/his platform. She/he should find a user of the same platform who is *new* to the installation (or relatively so, after all we want everyone to be using it 8^). You learn *important* things from a pair of fresh eyes. The newbie should also be committed to installing and testing, collaborating with the primary contact as required, who can shuttle information back to the development team. 3) Build libraries of test applications to exercise PHP as thoroughly as possible. Develop test modules in conjunction with new features. What we need is something akin to javadoc in which the two activities are intrinsically related. 4) Discuss strategies to improve response to trouble reports on the list(s). The current method of "pick what interests you" seems pretty haphazard. Perhaps organizing the incoming stuff by platform/OS and dividing them up chronologically? The primary platform contacts might serve as the first line of response, or have one or more colleagues handling that. Perhaps you could get additional folks to vet issues coming in for the "minor" platforms? (Stanislav Malyshev) Create automated tools and checklists for testing RCs. (Rasmus) I would much rather start a process by which we never release something until the release candidate has passed a checklist. Something like:

	Apache/Linux/Mysql                    y y y
	Apache/Linux/pgsql                    y y y
	Apache/Linux/Oracle                   y y y
	Apache/FreeBSD/MySQL                  y y y
	Apache/FreeBSD/pgsql                  y y y
	IIS-CGI/Win32/MSSQL                   y y y
	IIS-CGI/Win32/Oracle                  y y y
	IIS-ISAPI/Win32/MSSQL                 y y y
	IIS-ISAPI/Win32/Oracle                y y y

Any code changes would require the RC to be rebuilt and for this checklist to be cleared. The release should then be generated from the RC CVS tag so the release is exactly like the RC that passed. (Rick Rodgers) Add Solarix, UIX? (Andrei) Have a chart of which modules are currently stable and which are not (sort of like Tinderbox in mozilla.org). Regression test suite is sorely missed too. (Rasmus) Rework old testing system from PHP 2/3 (Sascha) Let multiple people take care of one target. Have a framework to run regression tests and report failures. I can supply the necessary scripts for automated

    builds.

(Jason) Setup a list with target configurations and assign a tester to a certain config. We could keep a list of each bug fix that's applicable to each config and when it comes time to test the RC, give each group their list of bugs. They could run appropriate regression tests and try to reproduce the bugs and then report. (Zeev) Zeev points out that QA is tedious, thankless and unispiring work... -- perhaps because of this, we should make sure that the QA team understands how important this kind of work is! (Thomas Swan) Use base new tests on existing tests. Examine the GCC testing process (Zeev) Zeev pointed out that this has been discussed before. Zeev -- Where is the previous work that has been done? It should help us out quite a good deal! Zak Greant Creative Director Nucleus Information Service Inc. "A camel is a horse designed by a committee." - Alex Issigonis (Designer of the Mini)