Re: [RFC] Facility for QA reports

On Wed, Dec 05, 2001 at 11:32:10AM -0600, Richard Lynch wrote : > GREAT RFC!!! Thanks, kudus also go to Jan who came up with most of the text and DB layout. > > > DROP TABLE IF EXISTS configure_arguments; > > CREATE TABLE configure_arguments ( > > id smallint(5) unsigned default NULL auto_increment, > > Thesis: > There is no real-world upper bound on the number of testers/tests > that could generate these records over time. > There could, some day, be more then smallint of these. As said in one of my previous mails, just ignore this smallint fuzz. I'm doing this for now too. > > configure text NOT NULL, > > > > PRIMARY KEY (id) > > ) TYPE=MyISAM; > > > # The actual testcase. It has an associated module_id and a > > # name. Beware that its 'id' is NOT simply auto increment > > # because this id is also references in an external entity > > # (in this case, the file in the php4 source tree which does > > # the testing). See at the bottom for [1] how to get this > > # numbers. > > I'd have gone for an indexed char field of "filename" or whatever. > > Actually, I'd have external reference to a "testfile" table with a > testfile_id to provide external reference. See below. > > I'm not quite grasping what the ID you are proposing *IS* though... Rather then referencing an external filename, the externel testcase file has to reference the particular ID of its testcase. We may want to change the filenames for whatever convention which would result that we need to update the DB do. I hate non-self-maintaining systems. > It would be better, I think, to have a table of the known "testfile" > filenames, with their relative path, any bug IDs they are alleged to > test, an author perhaps, and a module they are related to, as well as > (possibly NULL) distro, OS, etc that they are alleged to be closely > tied to. > > IE, if this particular "testfile" tests GD on PalmPilots (maybe some > day, eh?) then OS would be non-NULL. But if it just tests GD's > ability to make a JPEG, platform-independent (to the best of our > knowledge) OS would be NULL. As you can see below, testcases itself can be assigned to bugs. Originally I thought this would be quite clever because it allows us to assign bugs at a finer grain. You have for example a bug which is only present for a certani report under a certain operating system. However, this system is not be meant as the ultimate interchange to connect everything together, so the its considered as a nice feature to add bug references to testcases. > > > DROP TABLE IF EXISTS testrun; > > CREATE TABLE testrun ( > > id smallint(5) unsigned default NULL auto_increment, > > user_id smallint(5) unsigned NOT NULL, > > os_id smallint(5) unsigned NOT NULL, > > os_version_id smallint(5) unsigned NOT NULL, > > distribution_id smallint(5) unsigned NOT NULL, > > configure_id smallint(5) unsigned NOT NULL, > > > > PRIMARY KEY (id) > > ) TYPE=MyISAM; > > > # testresult if and only IF it has failed. They're used to > > # build the links from the QA reports to the bug database. It > > # is not yet decided how this should be properly handled. > > A given testfile record would be optionally tied to a bug ID. Many > testfile records (and their corresponding tests) are a direct result > of bug-resolution. > > The success/failure of a given testfile record would be tied back to > the bug ID automatically. That is what is proposed. The 'bugs' textfield contains a comma-or-whatever separated list of associated bug reprots. > > > DROP TABLE IF EXISTS testresults; > > CREATE TABLE testresults ( > > id smallint(5) unsigned NOT NULL default '0', > > testrun_id smallint(5) unsigned NOT NULL, > > testcase_id smallint(5) unsigned NOT NULL, > > result char(80) NOT NULL, > > remark char(80) NOT NULL default '', > > bugs text NOT NULL default '', > > > > PRIMARY KEY (id) > > ) TYPE=MyISAM; > > > > The ./configure line should ONLY BE TAKEN FROM > > php4/config.nice to make it easier for parsing. Test reports > > which give problems on parsing this won't make it into the > > database ! After all, this isn't a real problem isn't? Just > > copy&paste over the damn content of the file. > > Perhaps I'm foolish, but I was envisioning a "make test" which would > automagically log-in and *DO* all the database insertions... This is also mentioned at the very bottom. Originally it was not the primary goal to archive (too much automation can be bad -> m$). However if people tend to be very lazy I'm fine with doing this. I'll let others decide this. > And would not even require that the "make test" be done by a QA person, per > se. Everyyone submitting qa reprots need a qa account.Either an already existing CVS account or a qa-report only account. This is discussed in detail in the RFC. > It would also create a single text file that had everything needed to > be pasted into a giant box to be parsed on-line, in the event that a > given test box was not wired to the 'net, and the "make test" would > report failure to connect in such a way that it was painfully obvious > to the tester that the given file should be fed to the given URL. I see some problems getting all the details accurately (for example, linux distributions. And, it _does_ matter which distributions as they're all different in terms of library versions and let expirienced users spot possible bugs earlier, for example due a known lib problem ). > > cat config.nice, for example, would be used to add to this report > file, and then you wouldn't need to worry about those pesky humans > messing things up :-) Yes, config.nice is used for cofnigure parsing. > So, what I was envisioning is a "make test" where an idiot wouldn't > even need to know what all the tests meant, and the output would > automatically report back to this db which "testfile"s failed, which > are directly related to bug IDs, modules, OSes, and suchlike, so that > the developers and/or QAers can use them. > > IE, *every* end-user we induce to do "make test" would be adding data > to the database for us to know what test cases break on what machines > post-release. I see the problem this service could be abused if we don't require registration (and after all, registration is required to tie multiple reports together to a person who did them all so we can check back with him for particular problems). > I dunno how many end-users we can get to do it, and they'd have to be > made aware that we're reporting back info for privacy issues, > though... > > In an ideal world, the "make test" and the "testfile" scripts would > be good enough to winnow out stupid reports, where, say, GD failed > since GD wasn't installed, and only report meaningful data: > GD installed, works. > GD installed, failed. > (with all the OS and whatnot anybody could desire to figure out the pattern) > > Since the PHP version will be included, and since QA-ers will be the > ones doing the RC reports (in theory) we'll easily be able to winnow > out any old chaff of those pesky end-user reports from ancient > history (last week) and focus on the RC. > > Still, having a *HUGE* body of "make test" output from every end-user > would be invaluable, IMHO, to once-and-for-all tracking down bug > dependencies. > > General Overview: > Automate "make test" so that: > Any idiot can generate useful, automatically-parsable results. > Said results are, by default, reported back automatically, but with > switch to suppress reporting. > Said results are always dumped to a file that end-user can edit to > mask out whatever they dis-liked us knowing, and then paste into a > URL. > > Yes, we'll have umpteen thousand "GD works on Linux" reports. It's > worth it for the pattern analysis of where GD does *NOT* work. > > Maybe I'm being naive, or idealistic, though... - Markus -- Please always Cc to me when replying to me on the lists.