Re: Travis CI unit tests

On Sat, Apr 5, 2014 at 8:41 PM, Stas Malyshev <smalyshev@sugarcrm.com>wrote: > Hi! > > As many of us know, we have a setup on Travis CI which automatically > runs unit tests for our active branches. This is very convenient for > evaluating pull request and status of the branches and keeping us from > making changes that break something without us noticing. However, > there's a problem with this setup, and the problem is that CI build is > never kept in the green for more than a a few days, the "normal" state > of it is always failing. I think this situation is not normal and, > frankly, embarrassing for the project, and propose to institute a policy > to change it. > The start would be recognizing that the goal is to keep the CI tests > green. Once they are green, and somebody commits a change that makes > them fail, we have the following options (note I'm not advocating for > now any of them, just enumerating them) to fix it: > > 1. Revert the change immediately, and ask the submitter to fix the pull > (pulls can be tested too on CI too) and re-submit it when it's green. > > 2. Wait for the change developer for a reasonable short time to fix the > tests, if that does not happen, revert the change. > > 3. Put the failing tests into XFAIL until they are fixed by somebody. > > 4. Have somebody - e.g. RM for the branch - to modify the failing test > to reflect the new results. > > These options, of course, can be used in combination and in > case-per-case basis, but I think we need some explicit attention to the > topic. I welcome everybody to propose suggestions about other options or > ways we could fix this situation going forward and would like to hear > the ideas about how we could make Travis CI tests useful and passing > successfully, thus ensuring better quality in the project. After the > initial discussion, I plan to write an RFC summarizing the proposals and > have it voted on and, hopefully, accepted, so we'd have official policy > on it, but fir now I'd also like to raise awareness of the issue and > solicit ideas and participation of the members of the project on this > topic. > It would be a good start to send a mail to the person introducing a CI build failure automatically. I think to the most part people are simply not aware that they introduce failures - it's not like anybody goes on travis half an hour after doing a commit to check whether everything works fine. Maybe bugging people about this is already enough to keep a green build most of the time? Nikita