Re: [RFC] Facility for QA reports

I'm just a lurker here, but I thought I'd toss in my bit... I would be happy to help on the design/implementation of this if some help is required. phpBugTracker (http://phpbt.sourceforge.net/) has been my time waster as of late, but this looks like fun. One thing that caught my eye was the auto-submission of test results -- I have a fair amount of experience with XML-RPC and PHP, which could be a great fit for that functionality. On Wed, Dec 05, 2001 at 12:29:06PM +0100, Markus Fischer wrote: > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > This is a proposal initiated by me and written down by Jan > Lehnardt and me. > > We are asking for comments on this, generally if others see > sense in what we propose and more input on planning the > details. > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > RFC useful test results: > ======================== > > by: Markus Fischer <mfischer@guru.josefine.at> > and Jan Lehnardt <jan@lehnardt.de> > > This RFC covers the topic how 'make test' results can become > more useful to the PHP-QA Team and the PHP developers. > > > Current situation: > ------------------ > Currently the 'make test' results are simply posted to > php-qa@lists.php.net or are posted to the new Wiki. Sometimes > along with the according './configure' line and/or the 'uname > -a' result. Usually the reports are not in a consistent > structure. The Developers themselves have to look for failed > test and track them down to a proper bug report. This shall > not insult the activity of the QA team in any way, but we > think the test results can be improved. > > > What to do? > ----------- > How can we improve the usefulness of the 'make test' results? > > > Proposal: > --------- > We set up a system for collecting the results on qa.php.net. > They are stored in a database, split up by user, operation > system, OS version, OS distribution, PHP module and testcase. > > > Intention: > ---------- > The relational storage of the data coming with a test results > allows other QA team members and the PHP developers to > quickly look up failing testcases for any entity (e.g. OS, > PHP module). Failed test cases can be assigned to a bug > report on qa.php.net and/or a PHP developer (like the bugs > themselves). > > > > Some implementation details: > ============================ > > User Authentication: > -------------------- > > User authentication is needed so users can update their > reports and add can add additional reports. A 'user' can > submit different reports (different environments for example) > and also update existing reports (updating is meant to > overwrite the existing one, not a field by field editing > capability). > > Due the fact that some members of the QA team have an > existing CVS account and some don't (and it would be no good > idea to give out CVS accounts just to submit test reports) a > second, additional, authentication system will be introduced. > > Before submitting a testreport, a user has to be logged in > (obviously). There is ONE login area and ONE registration > area. If the user has an existing CVS account the user > doesn't need to register a QA accuont. His CVS > username/password will be happily accepted. > > That means, QA accounts are ONLY for people not having a CVS > accuont. If you apply for a QA account (which is done > automatically) you enter your usual things like > user/pass/pass2/email . This username is ALWAYS prepended > with 'qa_' to minimize a future clash of a CVS account > username and a QA account name. > > Following issues not yet targed: > A user used his CVS account for his reports but his access has > been canceled. This will lead to administrative action then. > > > The web frontend: > ================= > > Viewing submitted reports (of course no login required): > -------------------------------------------------------- > > No actual steps have been taken yet to 'design' the frontend. > Basically, like bugs.php.net, it allows filtering the > submited test runs (most likely for failed tests ;). > > The optional association of a testresult with bug reports > will cause an online link. > > Due to consistency OS Versions and Distributions can be > selected from a predefined list. With that we try to solve > the problem that we will have FreeBSD 4.4 and FreeBSD v4.4 as > two concurrent OS types, simply because the users chose a > diffenrent input. > > > Submitted a new test report (you have to be logged in): > ------------------------------------------------------- > > The submit page is split in to parts: first you only choose > your operating system and on the next page you enter the > version number, the distribution and submit all the > necessary information (not yet decided how this is done. > Simple text area or file uploads). Distributions can of course > be varios linux distributions as well as release tags (e.g. > FreeBSD 4.4-RELEASE and FreeBSD 4.4-STABLE. > > > Database tables: > ================ > > This is a draft layout and is subject to change. Description > of each table is held within their comments. > > # > # Table structure for table 'configure_arguments' > # > # 'configure' is simply the full configure line and is > # taken straight from the php4/config.nice file but with > # stripped off quotes and newlines and comments removed. > > DROP TABLE IF EXISTS configure_arguments; > CREATE TABLE configure_arguments ( > id smallint(5) unsigned default NULL auto_increment, > configure text NOT NULL, > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > > # > # Table structure for table 'os' > # > # Operating Systems. Contains the main operation system > # names against which a test can be ran. This includes names > # like 'Linux' and 'FreeBSD' (or whatever their correct > # spelling is ;) > > DROP TABLE IF EXISTS os; > CREATE TABLE os ( > id smallint(5) unsigned default NULL auto_increment, > name char(40) NOT NULL, > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > # > # Table structure for table 'os_versions' > # > # Version of the operating system. os_id obviously equals the > # id from table os. Version is the version string, for > # example '2.2.14' or '4.4'. It's based on a per-os basis. > > DROP TABLE IF EXISTS os_versions; > CREATE TABLE os_versions ( > id smallint(5) unsigned default NULL auto_increment, > os_id smallint(5) unsigned NOT NULL, > version char(40) NOT NULL, > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > # > # Table structure for table 'distributions' > # > # This is only made for Linux so far but since it has a > # relation to os_id it can be extended. Linux kernel version > # alone don't tell much. Therefore we cover ALL available > # distribution, e.g. 'Debian Potato, Debian Unstable, > # Mandrake xx, Redhat yy, SuSE xx' > > DROP TABLE IF EXISTS distributions; > CREATE TABLE distributions ( > id smallint(5) unsigned default NULL auto_increment, > os_id smallint(5) unsigned NOT NULL, > distribution char(80) NOT NULL, > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > # > # Table structure for table 'modules' > # > # 'modules' are the extension to PHP, e.g. ext/curl, ext/wddx > # but also ext/standard of course. Although every testcase > # (see below) can be uniquely identified they have to be > # categorized of course. > > DROP TABLE IF EXISTS modules; > CREATE TABLE modules ( > id smallint(5) unsigned default NULL auto_increment, > name char(80) NOT NULL, > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > # > # Table structure for table 'testcase' > # > # The actual testcase. It has an associated module_id and a > # name. Beware that its 'id' is NOT simply auto increment > # because this id is also references in an external entity > # (in this case, the file in the php4 source tree which does > # the testing). See at the bottom for [1] how to get this > # numbers. > > DROP TABLE IF EXISTS testcase; > CREATE TABLE testcase ( > id int NOT NULL, > module_id smallint(5) unsigned NOT NULL default '0', > name char(80) NOT NULL, > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > # > # Table structure for table 'testrun' > # > # Practically, this is the table which glues together > # everything. A combination of a user with his > # os,version,distribution runs his configure command and all > # testresults are identified with this testrun.id (and also > # their respective testcase) > > DROP TABLE IF EXISTS testrun; > CREATE TABLE testrun ( > id smallint(5) unsigned default NULL auto_increment, > user_id smallint(5) unsigned NOT NULL, > os_id smallint(5) unsigned NOT NULL, > os_version_id smallint(5) unsigned NOT NULL, > distribution_id smallint(5) unsigned NOT NULL, > configure_id smallint(5) unsigned NOT NULL, > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > # > # Table structure for table 'testresults' > # > # The information we actually want. result is either > # 'failed', 'passed' or 'skipped'. I also see the > # possibility of using a SET but found it less flexible. The > # remark is actually a generated output (optionally) by the > # testscript matching the testcase. > # > # The 'bug' field contains a comma separator list of > # associated bug report numbers for this particular > # testresult if and only IF it has failed. They're used to > # build the links from the QA reports to the bug database. It > # is not yet decided how this should be properly handled. > > DROP TABLE IF EXISTS testresults; > CREATE TABLE testresults ( > id smallint(5) unsigned NOT NULL default '0', > testrun_id smallint(5) unsigned NOT NULL, > testcase_id smallint(5) unsigned NOT NULL, > result char(80) NOT NULL, > remark char(80) NOT NULL default '', > bugs text NOT NULL default '', > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > # > # Table structure for table 'user' > # > # Information for created accounts. The 'name' field will be > # something like 'mariandl' BUT upon login she has to use > # 'qa_mariandl' (!). Why is explained above the database > # table description. > # > # The field 'cvsuser' is only filled out with users logged in > # from their CVS account which DO NOT have an QA test report > # account only. This prevents problems with people loosing > # their CVS account and hey, we also need the IDs for the > # testruns/cases ;) > > DROP TABLE IF EXISTS user; > CREATE TABLE user ( > id smallint(5) unsigned default NULL auto_increment, > name char(20) NOT NULL default '', > password char(20) NOT NULL default '', > email char(80) NOT NULL default '', > cvsuser char(80) NOT NULL default '', > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > > Each test case is identified by an unique id. The PQATCUIA > [1] leads the placing of this identifier. We add a new make > option like 'make testreport' which generates easily parsable > output for database insertion. (Later functionality could be > 'make testreportandsubmititdirectly' which directly sends all > needed data to qa.php.net). > > The ./configure line should ONLY BE TAKEN FROM > php4/config.nice to make it easier for parsing. Test reports > which give problems on parsing this won't make it into the > database ! After all, this isn't a real problem isn't? Just > copy&paste over the damn content of the file. > > > > [1] PhpQATestCaseUniquieIdentifierAssociation ;) > > -- > Please always Cc to me when replying to me on the lists. > RFC useful test results: > ======================== > > by: Markus Fischer <mfischer@guru.josefine.at> > and Jan Lehnardt <jan@lehnardt.de> > > This RFC covers the topic how 'make test' results can become > more useful to the PHP-QA Team and the PHP developers. > > > Current situation: > ------------------ > Currently the 'make test' results are simply posted to > php-qa@lists.php.net or are posted to the new Wiki. Sometimes > along with the according './configure' line and/or the 'uname > -a' result. Usually the reports are not in a consistent > structure. The Developers themselves have to look for failed > test and track them down to a proper bug report. This shall > not insult the activity of the QA team in any way, but we > think the test results can be improved. > > > What to do? > ----------- > How can we improve the usefulness of the 'make test' results? > > > Proposal: > --------- > We set up a system for collecting the results on qa.php.net. > They are stored in a database, split up by user, operation > system, OS version, OS distribution, PHP module and testcase. > > > Intention: > ---------- > The relational storage of the data coming with a test results > allows other QA team members and the PHP developers to > quickly look up failing testcases for any entity (e.g. OS, > PHP module). Failed test cases can be assigned to a bug > report on qa.php.net and/or a PHP developer (like the bugs > themselves). > > > > Some implementation details: > ============================ > > User Authentication: > -------------------- > > User authentication is needed so users can update their > reports and add can add additional reports. A 'user' can > submit different reports (different environments for example) > and also update existing reports (updating is meant to > overwrite the existing one, not a field by field editing > capability). > > Due the fact that some members of the QA team have an > existing CVS account and some don't (and it would be no good > idea to give out CVS accounts just to submit test reports) a > second, additional, authentication system will be introduced. > > Before submitting a testreport, a user has to be logged in > (obviously). There is ONE login area and ONE registration > area. If the user has an existing CVS account the user > doesn't need to register a QA accuont. His CVS > username/password will be happily accepted. > > That means, QA accounts are ONLY for people not having a CVS > accuont. If you apply for a QA account (which is done > automatically) you enter your usual things like > user/pass/pass2/email . This username is ALWAYS prepended > with 'qa_' to minimize a future clash of a CVS account > username and a QA account name. > > Following issues not yet targed: > A user used his CVS account for his reports but his access has > been canceled. This will lead to administrative action then. > > > The web frontend: > ================= > > Viewing submitted reports (of course no login required): > -------------------------------------------------------- > > No actual steps have been taken yet to 'design' the frontend. > Basically, like bugs.php.net, it allows filtering the > submited test runs (most likely for failed tests ;). > > The optional association of a testresult with bug reports > will cause an online link. > > Due to consistency OS Versions and Distributions can be > selected from a predefined list. With that we try to solve > the problem that we will have FreeBSD 4.4 and FreeBSD v4.4 as > two concurrent OS types, simply because the users chose a > diffenrent input. > > > Submitted a new test report (you have to be logged in): > ------------------------------------------------------- > > The submit page is split in to parts: first you only choose > your operating system and on the next page you enter the > version number, the distribution and submit all the > necessary information (not yet decided how this is done. > Simple text area or file uploads). Distributions can of course > be varios linux distributions as well as release tags (e.g. > FreeBSD 4.4-RELEASE and FreeBSD 4.4-STABLE. > > > Database tables: > ================ > > This is a draft layout and is subject to change. Description > of each table is held within their comments. > > # > # Table structure for table 'configure_arguments' > # > # 'configure' is simply the full configure line and is > # taken straight from the php4/config.nice file but with > # stripped off quotes and newlines and comments removed. > > DROP TABLE IF EXISTS configure_arguments; > CREATE TABLE configure_arguments ( > id smallint(5) unsigned default NULL auto_increment, > configure text NOT NULL, > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > > # > # Table structure for table 'os' > # > # Operating Systems. Contains the main operation system > # names against which a test can be ran. This includes names > # like 'Linux' and 'FreeBSD' (or whatever their correct > # spelling is ;) > > DROP TABLE IF EXISTS os; > CREATE TABLE os ( > id smallint(5) unsigned default NULL auto_increment, > name char(40) NOT NULL, > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > # > # Table structure for table 'os_versions' > # > # Version of the operating system. os_id obviously equals the > # id from table os. Version is the version string, for > # example '2.2.14' or '4.4'. It's based on a per-os basis. > > DROP TABLE IF EXISTS os_versions; > CREATE TABLE os_versions ( > id smallint(5) unsigned default NULL auto_increment, > os_id smallint(5) unsigned NOT NULL, > version char(40) NOT NULL, > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > # > # Table structure for table 'distributions' > # > # This is only made for Linux so far but since it has a > # relation to os_id it can be extended. Linux kernel version > # alone don't tell much. Therefore we cover ALL available > # distribution, e.g. 'Debian Potato, Debian Unstable, > # Mandrake xx, Redhat yy, SuSE xx' > > DROP TABLE IF EXISTS distributions; > CREATE TABLE distributions ( > id smallint(5) unsigned default NULL auto_increment, > os_id smallint(5) unsigned NOT NULL, > distribution char(80) NOT NULL, > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > # > # Table structure for table 'modules' > # > # 'modules' are the extension to PHP, e.g. ext/curl, ext/wddx > # but also ext/standard of course. Although every testcase > # (see below) can be uniquely identified they have to be > # categorized of course. > > DROP TABLE IF EXISTS modules; > CREATE TABLE modules ( > id smallint(5) unsigned default NULL auto_increment, > name char(80) NOT NULL, > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > # > # Table structure for table 'testcase' > # > # The actual testcase. It has an associated module_id and a > # name. Beware that its 'id' is NOT simply auto increment > # because this id is also references in an external entity > # (in this case, the file in the php4 source tree which does > # the testing). See at the bottom for [1] how to get this > # numbers. > > DROP TABLE IF EXISTS testcase; > CREATE TABLE testcase ( > id int NOT NULL, > module_id smallint(5) unsigned NOT NULL default '0', > name char(80) NOT NULL, > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > # > # Table structure for table 'testrun' > # > # Practically, this is the table which glues together > # everything. A combination of a user with his > # os,version,distribution runs his configure command and all > # testresults are identified with this testrun.id (and also > # their respective testcase) > > DROP TABLE IF EXISTS testrun; > CREATE TABLE testrun ( > id smallint(5) unsigned default NULL auto_increment, > user_id smallint(5) unsigned NOT NULL, > os_id smallint(5) unsigned NOT NULL, > os_version_id smallint(5) unsigned NOT NULL, > distribution_id smallint(5) unsigned NOT NULL, > configure_id smallint(5) unsigned NOT NULL, > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > # > # Table structure for table 'testresults' > # > # The information we actually want. result is either > # 'failed', 'passed' or 'skipped'. I also see the > # possibility of using a SET but found it less flexible. The > # remark is actually a generated output (optionally) by the > # testscript matching the testcase. > # > # The 'bug' field contains a comma separator list of > # associated bug report numbers for this particular > # testresult if and only IF it has failed. They're used to > # build the links from the QA reports to the bug database. It > # is not yet decided how this should be properly handled. > > DROP TABLE IF EXISTS testresults; > CREATE TABLE testresults ( > id smallint(5) unsigned NOT NULL default '0', > testrun_id smallint(5) unsigned NOT NULL, > testcase_id smallint(5) unsigned NOT NULL, > result char(80) NOT NULL, > remark char(80) NOT NULL default '', > bugs text NOT NULL default '', > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > # > # Table structure for table 'user' > # > # Information for created accounts. The 'name' field will be > # something like 'mariandl' BUT upon login she has to use > # 'qa_mariandl' (!). Why is explained above the database > # table description. > # > # The field 'cvsuser' is only filled out with users logged in > # from their CVS account which DO NOT have an QA test report > # account only. This prevents problems with people loosing > # their CVS account and hey, we also need the IDs for the > # testruns/cases ;) > > DROP TABLE IF EXISTS user; > CREATE TABLE user ( > id smallint(5) unsigned default NULL auto_increment, > name char(20) NOT NULL default '', > password char(20) NOT NULL default '', > email char(80) NOT NULL default '', > cvsuser char(80) NOT NULL default '', > > PRIMARY KEY (id) > ) TYPE=MyISAM; > > > Each test case is identified by an unique id. The PQATCUIA > [1] leads the placing of this identifier. We add a new make > option like 'make testreport' which generates easily parsable > output for database insertion. (Later functionality could be > 'make testreportandsubmititdirectly' which directly sends all > needed data to qa.php.net). > > The ./configure line should ONLY BE TAKEN FROM > php4/config.nice to make it easier for parsing. Test reports > which give problems on parsing this won't make it into the > database ! After all, this isn't a real problem isn't? Just > copy&paste over the damn content of the file. > > > > [1] PhpQATestCaseUniquieIdentifierAssociation ;) > > -- > PHP Quality Assurance Mailing List <http://www.php.net/> > To unsubscribe, e-mail: php-qa-unsubscribe@lists.php.net > For additional commands, e-mail: php-qa-help@lists.php.net > To contact the list administrators, e-mail: php-list-admin@lists.php.net