Re: [VOTE] CI tests RFC

On Fri, Apr 25, 2014 at 8:41 PM, Stas Malyshev <smalyshev@sugarcrm.com>wrote: > Hi! > > > Would be nice if we could also create some policy regarding the usage of > > XFAILs (when should be a failing test marked as XFAIL, should we add the > > tests from the open bugreports as XFAIL by default, whose responsibility > > is to make sure that it will be fixed eventually, etc.). > > I agree. I didn't get into this but it definitely makes sense to have > some rules there. > yeah, I'm glad that we managed to reach a state where we have 0/close-to-zero number of failing tests for the default builds on the major platforms, but the number of XFAILS are somewhat scarry, and some of there are pretty old too (there are a couple of them which I remember discussed to be fixed before 5.4.0 final...). the reason why we introduced the XFAILs were to be able to distinguish between the old/known/low-priority problems from new/important test failures. but now if we want to keep the failing test numbers on zero, sooner or later people will start putting XFAILs on everything to satisfy the CI, and we will end up in the same situation where the less important XFAILS will cause the important ones to slip under the radar. > > > I would also like to extend the current travis config a bit (we could > > have more exts, more axes for stuff like ts/nts builds, enable debug > > builds, so memory leaks are also triggering the test failures, etc.), > > You're more than welcome :) I've planned to get to some of it next - > i.e. going through the list of exts and see which ones we can support on > Travis - but any help would be great. > I've created a PR with some travis related changes: https://github.com/php/php-src/pull/654 I've also looked into having both 32 and 64 bit builds, but it seems travis only supports 64bit platforms atm: https://github.com/travis-ci/travis-ci/issues/986 and also looked into enabling the email notifications, but it seems that it is a bit complicated: http://docs.travis-ci.com/user/notifications/#Email-notifications the default behavior is: "By default, email notifications will be sent to the committer and the commit author, if they are members of the repository (that is, they have push or admin permissions for public repositories, or if they have pull, push or admin permissions for private repositories). ... By default, a build email is sent to the committer and the author, but only if they have access to the repository the commit was pushed to. This prevents forks active on Travis CI from notifying the upstream repository's owners when they're pushing any upstream changes to their fork. It also prevents build notifications from going to folks not registered on Travis CI. ... The email address is then determined based on the email address in the commit, but only if it matches one of the email addresses in our database. We synchronize all your email addresses from GitHub, solely for the purpose of build notifications." which we can't really use, because almost nobody has push/admin rights to our github repo, and adding an explicit email address (like internals@ or php-qa@) instead of using the default strategy would mean that any fork which get's activated on travis will start spamming those lists. :/ -- Ferenc Kovács @Tyr43l - http://tyrael.hu