Re: [VOTE] CI tests RFC

On Sat, Apr 26, 2014 at 8:51 AM, Joe Watkins <pthreads@pthreads.org> wrote: > On Fri, 2014-04-25 at 11:41 -0700, Stas Malyshev wrote: > > Hi! > > > > > Would be nice if we could also create some policy regarding the usage > of > > > XFAILs (when should be a failing test marked as XFAIL, should we add > the > > > tests from the open bugreports as XFAIL by default, whose > responsibility > > > is to make sure that it will be fixed eventually, etc.). > > > > I agree. I didn't get into this but it definitely makes sense to have > > some rules there. > > > > > I would also like to extend the current travis config a bit (we could > > > have more exts, more axes for stuff like ts/nts builds, enable debug > > > builds, so memory leaks are also triggering the test failures, etc.), > > > > You're more than welcome :) I've planned to get to some of it next - > > i.e. going through the list of exts and see which ones we can support on > > Travis - but any help would be great. > > > > > > -- > > Stanislav Malyshev, Software Architect > > SugarCRM: http://www.sugarcrm.com/ > > (408)454-6900 ext. 227 > > > > Morning Stas, > > I got time to read through this morning properly, obviously, yes > ... > but I'm not sure about the options, maybe I missed some relevant > conversation around it, not sure ... a few questions quick before I > finish voting if you don't mind ... > > Update test seems like questionable action, but see that you and > mike > voted for it, can I hear reasoning for that ? > > I don't see when there would be a good reason to retain a broken > commit > for a week, isn't that going to just render all of this pointless if > tests can fail for a week at a time ? > > Wouldn't it be better if all changes were done on branches, so > they can > be reviewed and integrated before being merged at all ? > > Maybe problematic for minor fixes but are they really the kind of > thing > that cause CI to bork ? > > Cheers > Joe > > I also agree that update test is a questionable option there. If we are talking about fixing a bad test, then it is obvious, and shouldn't be even put up for a vote, as naturally everybody including the RMs are free to fix the broken tests. So the only other reason listing that option is when somebody intentionally or unintentionally changes the behavior of some code without updating the tests. If that happens without proper discussion or RFC, I would say that it would be mandatory to revert such changes until consensus/approval of that can be made. tl;dr: fixing the wrong test is always an option, but to fix up the test to cover a bad code change shouldn't be an option. -- Ferenc Kovács @Tyr43l - http://tyrael.hu