RE: [PEAR-DEV] Re: PFC-RFC Update

Hi Toby, Any professionally-run QA organisation will normally consist of at least four separate activities. 1. Defining the quality goals. The criteria to be assessed should be clearly defined, and documented. Coding standards are part of this. Sensible coding standards are aimed at preventing defects, and ensuring sensible and consistent API design. Clearly defined *functional requirements* for each package are also essential. And *operational requirements* too. If you do not know what your package is supposed to do, then you cannot perform meaningful testing over time. Quality goals are updated periodically in response to factors such as trend analysis of reported bugs, improved engineering techniques, and (unfortunately) resistance from programmers. 2. Writing the test scripts. 'Script' here is a set of instructions for testing - not to be confused with code! It can be automated (and ideally should be), but it can also include manual inspection (correct license, coding standards followed, etc etc). The important point is this: without test scripts, your testing is not repeatable, and therefore is a waste of time. You also need predefined data sets, to help make testing as deterministic as possible. Imho, non-deterministic testing is not reliable enough for a credible QA effort. Automated test scripts should be written by the developers themselves, and reviewed by a QA team. Manual tests probably should be done by developers too, to scale better, and also reviewed by the QA team. It would be better if manual tests were implemented as some sort of questionnaire on the PEAR web site. 3. Running the automated tests. This should be a no-brain operation. The people doing this need access to the necessary hardware and software (inc. third-party stuff like Oracle!). They add value by actually running the tests, and sending the results back to the package maintainer. My experience from my own OpenSource project is that volunteers to do this are not hard to come by, for combinations of hardware and software that your users actually rely on. It's testers for the more esoteric combinations that prove to be more difficult, and to tackle this you probably want to talk to manufacturers about donations of licences etc etc. 4. Triage. When bugs are reported, they should be screened for duplicates, insufficient information, and downright stupidity. All bugs should be classified. Only screened bugs should then be forwarded on to the maintainer of a package to be worked on. Bug fixes should be accompanied by a new test to trap any future re-occurance of a bug. You'd be amazed at how often old bugs get re-introduced into software without this approach. Response times on bug fixing needs to be monitored, so that unmaintained packages can be identified and something done about it. This shouldn't be about beating up maintainers who are overworked in real life! -- As someone with managerial experience of running testing setups, I do not think that having a 'nominated panel' is the way to go. QA is not the province of a select few. It definitely is not something you bolt on the side, or something you farm out for others to do for you. To do it well, it has to be something that everyone buys into, and that everyone does. I'd recommend that the quality criteria is voted on by this list in the usual way. Establish the role of QA 'advocates' - named people who can be approached to provide QA assistance and advice to package maintainers. It's important that an 'advocate' does not actually implement the package, but works as an outside observer. A mailing list for advocates to share questions will help ensure a consistent approach over time, especially as advocates come and go. 'advocates' do not own QA, but they do police it. Oh, what the hell. If PEAR goes down an 'advocate' route, I'm willing to volunteer to be one such advocate. And to do anything else that'll help with the effort. Best regards, Stu -- > -----Original Message----- > From: Tobias Schlitt [mailto:toby@php.net] > Sent: 25 February 2003 18:57 > To: pear-dev@lists.php.net > Subject: [PEAR-DEV] Re: PFC-RFC Update > > > Hi y'all! > > I agree about PEAR needing some quality-assurance. But i > disagree the way it > should be done. I think the only division between PFC and > NON-PFC is too > glaring. > > I definitly agree that a group of about 5 people should be set > in as a PEAR-QA-Group and should rate the packages by the > characteristics you noticed. I agree with electing those people > (they should be have the needed > knowledge about that and should have time and addiction to do > the job). > > My Idea for rating the classes is to give them points (e.g. > stars) for matching some requirements. Initially every package > should get 1 point. When > fitting to the coding-standards it should get 2 points,... A > scale from 1 to > 5 is IMHO appropriate for that. > > What do you think about this system of rating? > > What about the way the classes are attempt to get rated? If > this 5 people in > QA-group have to rate every single release of a package they > will have a > huge amount of work to do. So, there should be a mechanism > implemented for > signing a new release as "to rate" or "same rate as last > release" (more or > less you should have the opportunity to mark a release as "to > rate" only > once per month or something else). > > Do you agree with that? > > The guidelines for each step have to be defined extremely clear > and viewable > for everyone out there (much more in detail than the > coding-standards are, i > think). > > So, i hope that some of you agree with my ideas. Flames to > /dev/null please. > > Regards, > Toby > -- > <?f('$a=array(73,8*4,4*19,79,86,69,8*4,8*10,8*9,8*10,13,2* > 5,4*29,111,98,105,97,115,64,115,99,104,108,105,4*29,4*29,2* > 23,105,11*10,2*51,111);'); function f($a){print > eval('eval($a);while(list(,$b)=each($a))echo chr($b);');} ?> > > -- > PEAR Development Mailing List (http://pear.php.net/) > To unsubscribe, visit: http://www.php.net/unsub.php >