PEAR Quality (vs. Quantity) RFC

according to the pear-meeting in Amsterdam i want to propose some points which might increase the transparency for the users to see how complete a package is. Summary ------- The main idea is to encourage developers to reach a high quality level and to show this transparently to the user, so she can decide on some hard facts which package to consider. For making this possible and to need as little man power as possible to enforce this some hard facts (automatic checks in this case) are needed. But quality is not all about hard facts. Things like OO-design, the use of Design Patterns, etc. might have an effect on the quality level of a package. Main focus ---------- This will try to describe a common process for automatically checking the hard facts for quality assurance in PEAR. This can be applied to the existing packages and to the proposed packages. Quality stages -------------- The following things can be automatically verified, in order to give a 'quality seal' (or whatever it is called) to a package. The seal consists of different checks as described below: * Stage 1 (minimal req.) - Proper inline documentation The package is checked for proper inline documentation, that means - every class, - every method, - every parameter, - every property has to be commented - the comments have to adhere to the phpDoc-rules To automatically check this it would be possible to use the phpDocumentor classes, i think. I dont know phpDocumentor that good, but i guess we can define something like 95% of the documentation have to be given to fulfill the requirements of this stage. * Stage 2 (minimal req.) - CS compliance The package should be checked for compliing to the PEAR-CS rules. If that is not the case the package should not even get accepted in PEAR. I would suggest that this stage always has to be fulfilled 100%. May be some kind of naming convention checking can be done too (use of the standard method names, such as 'connect', 'disconnect'). I don't know how since the check-script can't know what the package does, but may be someone has an idea. * Stage 3 (minimal req.) - Examples The package has to contain examples. To be able to automatically check this, there should be the definition that examples i.e. have to be in the directory 'examples' (i dont know if the current 'docs'-dir was meant for this purpose, and since everybody is putting a lot of things there, i think it makes more sense to define a new directory). The number of examples (number of files?) should depend on the size of the package. The bigger (LOC's or number of public methods) the package, the more examples should exist. Since the size of the API/LOC also (kind of) tells how complex the package is and how hard it is to use it. * Stage 4 - Unit tests Check for the existance of unit tests. Again, to be able to automatically check this they should be in the directory 'UnitTest'. Every public method should have it's own test class (that means also a seperate file). Then it is easy to automatically check the existance of the proper test. Example: The class contains the 3 public methods: getData, setData, removeData. Inside the directory 'UnitTest' there should be the following files: getData.php, setData.php, removeData.php Where each file does at least 4 tests for it's method. (why 4? one for the good case, second for failure, third and fourth for the upper and lower range checks) * Stage 5 - User documentation/Manual A pacakge should contain user documentation, in order to get starrted quickly. IMHO user documentation should also comply to a certain basic structure, such as that it has to have the following chapters: Executive summary (what's the package about, what can it do) Getting started (simple overview with simple examples for the most common uses) Detailed documentation and other chapters, the developer thinks are relevant To be able to automatically check for the proper existance I think the peardoc2 structure can be parsed/searched (please correct me here). * Stage 6 - UML diagrams A pacakge of high quality should provide UML diagrams for the users to easily overlook the entire structure of the package. This can be checked automatically, if we define to have a directory such as 'uml' where the diagrams have to be in. Since there are many tools to generate this (but no standard tool afaik which does round trip engineering for php code) this directory should also contain gif/jpg's of the diagrams. * Stage 7 - Design of the application This stage can not be verified automatically! The QA team or an selected team should check a package for it's OO-design and either suggest possible refactoring methods or approve the high quality of the package. This has to be done every time a new (major) release is done, so it seems like a lot of work. The pear website could now have an overview which shows the stages the package complies to. Since the first 5 (or 6) stages can be checked automatically, new package proposals can also be run through this test and the test can tell afterwards: the package complies to stage X with 50% etc. This way a quick overview can be given. May be this way we can also measure which package can apply for becoming a PFC. Please throw in ideas :-) -- Wolfram ... opensource @ vision:produktion ... http://opensource.visionp.de ... translating template engine .... http://sf.net/projects/simpletpl ... authentication system .... http://sf.net/projects/auth