Re: [RFC] Facility for QA reports

GREAT RFC!!!

    DROP TABLE IF EXISTS configure_arguments;
    CREATE TABLE configure_arguments (
      id smallint(5) unsigned default NULL auto_increment,

Thesis: There is no real-world upper bound on the number of testers/tests that could generate these records over time. There could, some day, be more then smallint of these. If there are never more than smallint of these, the wastage of a 32-bit int instead of smallint is negligible. Go for 32-bit int.

      configure text NOT NULL,

      PRIMARY KEY (id)
    ) TYPE=MyISAM;

    # The actual testcase. It has an associated module_id and a
    # name. Beware that its 'id' is NOT simply auto increment
    # because this id is also references in an external entity
    # (in this case, the file in the php4 source tree which does
    # the testing). See at the bottom for [1] how to get this
    # numbers.

I'd have gone for an indexed char field of "filename" or whatever. Actually, I'd have external reference to a "testfile" table with a testfile_id to provide external reference. See below. I'm not quite grasping what the ID you are proposing *IS* though... It would be better, I think, to have a table of the known "testfile" filenames, with their relative path, any bug IDs they are alleged to test, an author perhaps, and a module they are related to, as well as (possibly NULL) distro, OS, etc that they are alleged to be closely tied to. IE, if this particular "testfile" tests GD on PalmPilots (maybe some day, eh?) then OS would be non-NULL. But if it just tests GD's ability to make a JPEG, platform-independent (to the best of our knowledge) OS would be NULL.

    DROP TABLE IF EXISTS testrun;
    CREATE TABLE testrun (
      id smallint(5) unsigned default NULL auto_increment,

I'm seeing this break smallint some day as well... Same theses as above.

      user_id smallint(5) unsigned NOT NULL,
      os_id smallint(5) unsigned NOT NULL,
      os_version_id smallint(5) unsigned NOT NULL,
      distribution_id smallint(5) unsigned NOT NULL,
      configure_id smallint(5) unsigned NOT NULL,

      PRIMARY KEY (id)
    ) TYPE=MyISAM;

    # testresult if and only IF it has failed. They're used to
    # build the links from the QA reports to the bug database. It
    # is not yet decided how this should be properly handled.

A given testfile record would be optionally tied to a bug ID. Many testfile records (and their corresponding tests) are a direct result of bug-resolution. The success/failure of a given testfile record would be tied back to the bug ID automatically.

    DROP TABLE IF EXISTS testresults;
    CREATE TABLE testresults (
      id smallint(5) unsigned NOT NULL default '0',

int(11), see above. :-) And, of course, my int(11)'s all cascade through...

      testrun_id smallint(5) unsigned NOT NULL,
      testcase_id smallint(5) unsigned NOT NULL,
      result char(80) NOT NULL,
      remark char(80) NOT NULL default '',
      bugs text NOT NULL default '',

      PRIMARY KEY (id)
    ) TYPE=MyISAM;

    The ./configure line should ONLY BE TAKEN FROM
    php4/config.nice to make it easier for parsing. Test reports
    which give problems on parsing this won't make it into the
    database ! After all, this isn't a real problem isn't? Just
    copy&paste over the damn content of the file.

Perhaps I'm foolish, but I was envisioning a "make test" which would automagically log-in and *DO* all the database insertions... And would not even require that the "make test" be done by a QA person, per se. It would also create a single text file that had everything needed to be pasted into a giant box to be parsed on-line, in the event that a given test box was not wired to the 'net, and the "make test" would report failure to connect in such a way that it was painfully obvious to the tester that the given file should be fed to the given URL. cat config.nice, for example, would be used to add to this report file, and then you wouldn't need to worry about those pesky humans messing things up :-) So, what I was envisioning is a "make test" where an idiot wouldn't even need to know what all the tests meant, and the output would automatically report back to this db which "testfile"s failed, which are directly related to bug IDs, modules, OSes, and suchlike, so that the developers and/or QAers can use them. IE, *every* end-user we induce to do "make test" would be adding data to the database for us to know what test cases break on what machines post-release. I dunno how many end-users we can get to do it, and they'd have to be made aware that we're reporting back info for privacy issues, though... In an ideal world, the "make test" and the "testfile" scripts would be good enough to winnow out stupid reports, where, say, GD failed since GD wasn't installed, and only report meaningful data: GD installed, works. GD installed, failed. (with all the OS and whatnot anybody could desire to figure out the pattern) Since the PHP version will be included, and since QA-ers will be the ones doing the RC reports (in theory) we'll easily be able to winnow out any old chaff of those pesky end-user reports from ancient history (last week) and focus on the RC. Still, having a *HUGE* body of "make test" output from every end-user would be invaluable, IMHO, to once-and-for-all tracking down bug dependencies. General Overview: Automate "make test" so that: Any idiot can generate useful, automatically-parsable results. Said results are, by default, reported back automatically, but with switch to suppress reporting. Said results are always dumped to a file that end-user can edit to mask out whatever they dis-liked us knowing, and then paste into a URL. Yes, we'll have umpteen thousand "GD works on Linux" reports. It's worth it for the pattern analysis of where GD does *NOT* work. Maybe I'm being naive, or idealistic, though... -- WARNING richard@zend.com email address is an endangered species Use ceo@l-i-e.com instead