Re: PEAR Quality (vs. Quantity) RFC

Wolfram Kriesing wrote:

according to the pear-meeting in Amsterdam i want to propose some points which might increase the transparency for the users to see how complete a package is. Summary ------- The main idea is to encourage developers to reach a high quality level and to show this transparently to the user, so she can decide on some hard facts which package to consider. For making this possible and to need as little man power as possible to enforce this some hard facts (automatic checks in this case) are needed. But quality is not all about hard facts. Things like OO-design, the use of Design Patterns, etc. might have an effect on the quality level of a package. Main focus ---------- This will try to describe a common process for automatically checking the hard facts for quality assurance in PEAR. This can be applied to the existing packages and to the proposed packages. Quality stages -------------- The following things can be automatically verified, in order to give a 'quality seal' (or whatever it is called) to a package. The seal consists of different checks as described below: * Stage 1 (minimal req.) - Proper inline documentation The package is checked for proper inline documentation, that means - every class, - every method, - every parameter, - every property has to be commented - the comments have to adhere to the phpDoc-rules To automatically check this it would be possible to use the phpDocumentor classes, i think. I dont know phpDocumentor that good, but i guess we can define something like 95% of the documentation have to be given to fulfill the requirements of this stage.

Agreed, but you mention both phpDoc and phpDocumentor... it is my experience they are not the same thing, and that they can handle more than one format of apidoc, during the pear meeting, someone on IRC said phpDoc(umentor?) can support the ZDE style apidoc for example, which is what I use as standard and would carry on using to make my life easier (ZDE Auto-Complete uses the apidoc to show description, args and return for each method/class)

* Stage 2 (minimal req.) - CS compliance The package should be checked for compliing to the PEAR-CS rules. If that is not the case the package should not even get accepted in PEAR. I would suggest that this stage always has to be fulfilled 100%. May be some kind of naming convention checking can be done too (use of the standard method names, such as 'connect', 'disconnect'). I don't know how since the check-script can't know what the package does, but may be someone has an idea.

I think there are some checks that can be done automatically... like looking for tabs, is there any reason for there *ever* to be a tab in a PEAR file? But I think for the most part, we need someone, or the entire QA team to handle this.

* Stage 3 (minimal req.) - Examples The package has to contain examples. To be able to automatically check this, there should be the definition that examples i.e. have to be in the directory 'examples' (i dont know if the current 'docs'-dir was meant for this purpose, and since everybody is putting a lot of things there, i think it makes more sense to define a new directory). The number of examples (number of files?) should depend on the size of the package. The bigger (LOC's or number of public methods) the package, the more examples should exist. Since the size of the API/LOC also (kind of) tells how complex the package is and how hard it is to use it.

Is it a case of *must* have examples? or is the need for this negated when full docbook manual docs are written?

* Stage 4 - Unit tests Check for the existance of unit tests. Again, to be able to automatically check this they should be in the directory 'UnitTest'. Every public method should have it's own test class (that means also a seperate file). Then it is easy to automatically check the existance of the proper test. Example: The class contains the 3 public methods: getData, setData, removeData. Inside the directory 'UnitTest' there should be the following files:

    getData.php, setData.php, removeData.php
Where each file does at least 4 tests for it's method. (why 4? one for the good case, second for failure, third and fourth for the upper and lower range checks)

What do we do when, for example our class interacts with a database? Should we provide an MDB XML database definition file and then it could be created on the fly after upload to pearweb? then pearweb runs the unit tests... if we do this, unit tests will not work properly on users machines... unless we add something like this: ---- $ pear make-tests <package> <package> requires a database for its tests, please choose an option from those below: 1. Create Database and continue tests 2. Create Tables in existing Database 3. Skip Tests that require database 4. Skip all Tests Choose one: 1 Please enter a database name: foo Please enter a username: bar Please enter a password: baz ... Database generated, continuing tests <unit tests done here> All tests were successful, remove the database [Y/n]: Y Database removed. $ ---- You get the idea. this may not only mean additions to the pear command, but also phpUnit, package.xml (we could define what unit tests are there, whether or not they need a database and where the database schema is)

* Stage 5 - User documentation/Manual A pacakge should contain user documentation, in order to get starrted quickly. IMHO user documentation should also comply to a certain basic structure, such as that it has to have the following chapters:

    Executive summary (what's the package about, what can it do)
    Getting started (simple overview with simple examples for the
            most common uses)
    Detailed documentation
    and other chapters, the developer thinks are relevant
To be able to automatically check for the proper existance I think the peardoc2 structure can be parsed/searched (please correct me here).

Read back to my note on Stage 3

* Stage 6 - UML diagrams A pacakge of high quality should provide UML diagrams for the users to easily overlook the entire structure of the package. This can be checked automatically, if we define to have a directory such as 'uml' where the diagrams have to be in. Since there are many tools to generate this (but no standard tool afaik which does round trip engineering for php code) this directory should also contain gif/jpg's of the diagrams.

What about png? what point do these serve? will they be in the manual? displayed as ascii art on the command line? seems mostly pointless to me.

* Stage 7 - Design of the application This stage can not be verified automatically! The QA team or an selected team should check a package for it's OO-design and either suggest possible refactoring methods or approve the high quality of the package. This has to be done every time a new (major) release is done, so it seems like a lot of work.

I agree this is needed, you cannot automate this stuff.

The pear website could now have an overview which shows the stages the package complies to. Since the first 5 (or 6) stages can be checked automatically, new package proposals can also be run through this test and the test can tell afterwards: the package complies to stage X with 50% etc. This way a quick overview can be given. May be this way we can also measure which package can apply for becoming a PFC. Please throw in ideas :-)

At the end of the day, whatever is decided, ALL of these stages need excellent documentation. Examples of phpDoc(umentor) compatible apidoc (not really found a decent example anywhere), example of how to test apidoc using phpDoc(umentor). It should be also made clear what CS applies to example HTML, DocBook XML, etc... I think the main thing, is, examples, perhaps you could take an existing package and make it comply to all this, and document that process? (how about PEAR_Info, its nice and small! ::evil grin::) - Davey