[RFC] - How the QA Team will work. Rev 2.

Title: RFC - How the QA Team will work. ------------------------------------------------------ Author: helgi at trance dot is, smith at backendmedia dot com Revision: 2 Status: Active There has been talked about forming QA team, but no action yet taken. The following is proposal of how the QA team will work, what rights they have and so on. The Issues ----------- There is no official QA team for PEAR. There are only a few people who try to help with QA issues but they don't have any official status or rights to resolve issues themselves if needed. PEAR claims quality but that can't be met if there is no real QA team. PEAR group has been talking about fine tuning karma for all users so the current user got access to everything under /pear in cvs will be eliminated and thus QA needs special karma when that happens. More importantly there is currently no formal process or entity to handle orphaned packages or urgent fixes to packages. To summarize the problem * No QA team = can't claim Quality * Nobody has defined how the QA team is supposed to work, how it should be structured and what its rights (karma) are supposed to be. The Proposed Solution. ----------------------- small side note: Karma in this context doesn't just mean "technical" karma but also "political", meaning that they not only have the necessary access rights from a technical perspective (which currently most of the pear devs have anyways). They also have the right to commit to any part of the cvs if they deem it necessary (obviously following the necessary regulations) QA team membership: How the initial members are chosen: If there are more applicants than seats, there should be an open election, where everyone may cast as many votes as there are seats (only 1 vote may be cast per candidate). Of course, if there are less people than seats, all may be accepted and the remaining seats can be filled by QA group vote. Once the core team is formed all further team members can be voted in as per the rules stated below. There will be a QA core team with a limited number (no more than 7, with the pear group this should give a sufficient amount of people with enough karma that can react to bad releases) people with extended "core-QA-karma". In order for the QA core team to react quickly, the core QA team should be spread out across all time zones as good as possible. This rather small number should empower the QA team sufficiently to handle issues but doesn't trivialize access rights. Note: This issue needs more discussion since Klaus raised a good point in his post http://news.php.net/article.php?group=php.pear.dev&article=26756 but the problem is that it isn't wise to give many users so much karma. All other people interested are free to join the QA team. However potential new members for the QA team (core and non core) must be voted in by a 2/3 majority of the existing members of the QA team (core and non core). This is to prevent that the addition of new members is done even though there is a considerable amount of opposition, as this could lead to internal quarrels which could result in reducing the QA teams ability to efficiently address QA issues. Removing a member from either group requires a 2/3 majority as well. For casual helpers the QA mailing list will of course remain open. All members of the QA team (core and non core) are eligible to participate in QA team votes. Any member that leaves the QA team loses all his special karma which he may have received due to his membership. If any member of the core QA team leaves and as a result a time zone gets under staffed, QA should actively try to find a replacement. Notes: When voting for a new member voters must be sure that the person has been helping with bug fixes in the past or has the potentials to help with that. Voting: All votes (except membership related votes; see above) require a simple 50% majority. Any member of the QA team may call for a vote at any time. All voting is done on the QA mailing list. The core team may overrule any decision (this includes votes on memberships) of the QA team and may decide to hold their own internal vote to make the ultimate decision on how to proceed. Furthermore, the core QA team may also hold votes without even consulting the QA team. However this should be only done in rare cases. All members of the core QA team may make decision on their own if a decision is time critical (like pulling a broken release). However the core QA team should be aware that making too many decisions on their own may lead to alienating the other members and therefore if possible it should always be attempted to at least consult other members of the team. QA rights/responsibilities: Fixing bugs/making releases If any QA related issues are found in a package that QA has not been granted permission to change without consulting the maintainer,the QA team will file a bug report. If the issue remains unresolved for 1 month (2+1+1 weeks; see below) until QA may fix it themselves. If the issue isn't fix with in two weeks QA will email the maintainers about the issue (if the bug system won't have auto bug report notices every X week in the future) and if the problem isn't fixed within one week from that, QA will send yet another email to the maintainers and if no answer nor fix to the problem is provided within one week, all members of the core QA team will get the permission to modify any file needed to fix that QA issue as well as make a new release. If any major QA issues are found in any package and the QA team doesn't have permission to change that package without contacting the lead first then the QA team will file a bug report. Any major issues can stay open for 2 weeks (5+5+4 days; see below) until QA may fix it themselves. If the issue isn't fix within 5 days QA will email the maintainers about the issue (if the bug system won't have auto bug report notices every X week in the future) and if the problem isn't fixed with in 5 days, from that QA will send yet another email to the maintainers and if no answer nor fix to the problem is provided with in 4 days, all members of the core QA team will have permission to modify any file needed to fix that QA issue as well as make a new release. The same time frames apply when the QA team want to make a release because of unreleased fixes/improvements which are only available via CVS, or when the QA team wants to determine if a package has been orphaned (using the time frames specified for non major issues). The QA team can decide what are major issues after discussion on the QA Mailing-list. The QA team will for this reason keep track of those QA issues in their pear web subsection. This would only be for having overview over which package maintainers need to be emailed for reminders and such things. Orphaned Packages The QA team can use the above stated rules to determine if a package is orphaned. The core QA team gets lead rights on all orphaned packages until the original maintainer returns or a new maintainer is found. Deleting Package releases The core QA team also has the permission to delete any release that might have any issues that have been found after the release, to reduce the effects of the problem. This right must be exercised with caution. Package maintainers can also grant QA the rights to make a releases for their packages. PEAR web QA subsection Seems that there is a lack of QA material in the manual that people agree on and there is also need to make it move visible for people so they will actually follow those rules/guidelines. The QA team will therefore get its own QA subsection in the manual and will have control over it just like package maintainer controls package related docs in the manual. The QA team also gets its own subsection on pear web so people can more easily access QA material which is too dynamic or just related to the QA team own internal documentation purposes. There the QA will, for example publish their latest decisions. Furthermore here the QA team will be able to provide interfaces to QA team services. For example, here the QA team could offer an interface for maintainers to configure the access rights they want to grant the QA team. Finally the core QA team can add QA related notes to each package homepage to communicate QA related information to the user base. Approving first stable release The QA team needs to approve the first stable release of any major version. For this purpose maintainers upload the release to PEAR web and contact the QA team with a link to the release, then QA will decide if the release is ready or not. The core QA team will at this point approve the release trough PEAR web and release it for the maintainer. Notes: The QA team must always inform the current lead maintainers of any changes that are done to a given package using the currently configured email addresses on pear web. The core QA team may delegate tasks as much as possible to other QA team members. However no access rights will be assigned just for the sake of delegating a task (so members of the QA team may package a new release, but only members of the core QA team will have the necessary rights to upload the release). Summary of the solution QA core group access rights: * Max 5 core members that get the core-QA-karma. * the core QA team will get more permission to fix issues and make release of packages if reported problems are not fixed in a specific time frame * the core QA team assumes lead over orphaned packages * the core QA team gets the rights to delete releases of packages if it determines that the release has severe issues * the QA team is allowed to add QA related notes to package homepages * QA team needs to approve an first stable release for a given major version number (done by votes) * the entire QA team will identify QA problems, will help writing QA related documents, will help people resolve their QA problems and such things * More focus is needed on writing QA related things in the manual and make then more visible * Make a little QA corner at pear.php.net so QA is more visible for people and package devs are more aware of the QA team and how it works. * keep track of what QA does regarding packages that QA doesn't have permission to edit trough a wiki of some sort Actions Required if accepted. ----------------------------- * Forming of the QA team, electing the persons who should for the core of the QA team. * Giving the core dudes full CVS karma on the pear CVS module, as well as full access rights to package management part of pear web * Make that little QA pear page * Improve the QA manual pages * Finding out which packages lead allow QA to help keeping their package QA approved * Better coordination of the work which is done, who is going to do what. Know in advance who is going to write some more entries to the manual and so on. * Make a wiki page of some sort for the info tracking regarding QA editing package that QA doesn't have permission to edit. Change log ---------- Initial Release : 24 March 2004 Revision 1 : 26 March 2004 - Fixed some typos (left summarize since Alan did it that way so I can't see anything *wrong* with it) - added Lukas as author - added in "Issue" about pear group - added that more discussion is needed about core size - added about if member leaves the team - added small side note what the term Karma means in this RFC. - added regarding new member in the QA team. - No voting for pulling releases - How the initial members are chosen. - Changed name of section QA core group access right to QA rights/responsibilities - Made better sub sections for readability in QA rights/responsibilities - added how Orphaned Packages will be handled - More specific about Deleting Package releases - Reworked the wording of things in many places Revision 2: 6 April - Forgot to add the date of revision 1 ;/ - Spelling and grammar corrections provided by Klaus :) - Boosted core size to 7. - Changed the time frame regarding packages that haven't allowed QA to modify. - Changed how things are handled around first stable release Comments ---------- Klaus Guenther: Wants to boost up the core size to 10-15 members so the team can always react to any situation more quickly see http://news.php.net/article.php?group=php.pear.dev&article=26756 for more details. Also mentions that there is needed some clarification on what happens when a core member leaves the team, and that voting needs some time frame. Feels that QA could make use of PEPr for sending out emails to maintainers about bug reports and such. For Rev. 1 Klaus came with some ideas how orphan packages should be handled on the pear web. "That will require a new global maintainer: "PEAR QA", which would be merely a pseudo user. ......." (We leave the implementation details for a more advanced rights management solution to the pear web team) Alexey Borzov: Is against the core of QA, everything should be done in volunteer (isn't pear all about that?) also dislikes the voting idea, think person should only be accepted if he provides many quality bug fixes. packages should not be deleted if the case isn't extreme, if minor issue then down grade it's state. His entire comments can be read here http://news.php.net/article.php?group=php.pear.dev&article=26757 Daniel Convissor: Feels that time frame allowed for packages that haven't given QA the rights to modify them, be to long (Was fixed in Revision 2 )