Re: Re: Continuous Integration Servers for php-src

On Thu, Oct 15, 2009 at 10:27 PM, Eric Stewart <ericleestewart@gmail.com>wrote: > I have two thoughts on this and I'm not sure which is better. > > 1. We run a queue like explained by Isreal. All capable servers feed from > the queue. All non-capable servers are not included in the queue. Instead > they run on a scheduled basis but are still able to push test results to the > master. > > 2. A less sophisticated, but probably easier to develop solution would be > for each server to simply pull when possible instead of from a queue. Each > server would automatically check (say every 5 or 10 minutes) what the latest > SVN revision number is and if different than the last time it pulled, it > gets the updates then builds and tests. Once complete it pushes results back > to the master and then waits 5 or 10 minutes and pulls again. This option > might be easier to implement and would allow slower machines to build as > often as they can and faster machines will end up testing just about every > commit. > > I certainly don't want to discourage Isreal from his plans, but thought > this might be more optimal across a more heterogeneous infrastructure. It > will eliminate queue backups when a server goes offline for maintenance or > repair. It would eliminate the need for the master to register and keep > track of it's slaves. They come and go as they please. And the configuration > of the slaves might be less complicated. The slaves can simply identify > themselves by IP and OS-Version when they push their results. So the master > becomes more of a passive recipient to the slaves which would reduce > coupling and synchronization problems. > > Just my thoughts. > > Eric Lee Stewart > ericleestewart@gmail.com > > On Thu, Oct 15, 2009 at 7:26 PM, David Soria Parra < > David.Soriaparra@sun.com> wrote: > >> Hi Isreal, >> Hi Zoe, >> >> I'm not aware of the current status of the new test infrastructure and all >> the requirements. If it is possible to integrate the new test runner we >> should do this. It will significantly reduce the testing time on some >> machines (particular SPARC). >> >> I would also prefer that we are _not_ testing after every commit. A full >> compilation can take very long depending on the system. If we want to run >> tests on older systems (e.g. an old SPARC with Solaris 9) it will take ages >> and the queue will just be filled. It would suggest that we build max. 4 >> times a day to make sure that the queue can be processed. >> >> As soon as we have a working configuration for slaves I'll setup a slave >> on the OpenSolaris SPARC machine that php.net can use for testing >> purposes. >> >> David >> >> >> On 08.10.09 09:10, zoe wrote: >> >>> Hi Israel - this looks like a good initiative :-) Well done. >>> >>> I have a suggestion about the PHPT tests. We have been working on a new >>> test runner, it has major advantages over the existing test runner. It >>> runs tests in parallel - so depending on the numbers of processors you >>> have it can be much faster, the output is XML and would be easier for >>> you to handle than the existing test runner output. >>> >>> I would really like to see the new test runner integrated in what you >>> are doing. There is one problem, the implementation of REDIRECTTEST is >>> not done yet and I have no time to do it, however the code does handle >>> the fact that it's missing - the result is just that some (not a huge >>> number) of the tests are skipped. >>> >>> My only problem is that I have very little time currently to help with >>> this, I'm wondering if anyone on the wider QA team would be interested? >>> I can certainly help with answering any questions on how the >>> run-tests.php code works? >>> >>> Zoe >>> >>>> Hi everyone, >>>> >>>> My name is Israel Ekpo <iekpo@php.net >>>> <mailto:iekpo@php.net>> >>>> >>>> I am currently working on the UNIX buildbot system on the Debian >>>> server (80.82.121.134). >>>> >>>> Please review this implementation set up and let me know what you think. >>>> >>>> I just want to maintain transparent communication with all parties >>>> involved so that I don't end up >>>> straying too far from what is expected. >>>> >>>> After this continuous integration server is completed and finalized, >>>> it will be replicated >>>> across the other UNIX servers in a slave-master set up. >>>> >>>> For now, each slave will be doing pretty much what the master is doing >>>> (testing php 5.2, 5.3 and 6.0) >>>> >>>> However, the results will only be accessible and stored on the master >>>> server. >>>> >>>> Here is the rough draft for the implementation details: >>>> >>>> ==================================================================== >>>> 0. Target Subversion URLs >>>> ==================================================================== >>>> The priority as stated by Hannes Magnusson, is to set it up for >>>> PHP 5.3 first then PHP 5.2 and 6.0+ >>>> >>>> So the build bot will only be monitoring commits to the following >>>> subversion URLS : >>>> >>>> PHP 5.3: >>>> http://svn.php.net/repository/php/php-src/branches/PHP_5_3 >>>> PHP 5.2: >>>> http://svn.php.net/repository/php/php-src/branches/PHP_5_2 >>>> PHP 6: >>>> http://svn.php.net/repository/php/php-src/trunk >>>> >>>> A separate build daemon will be set up for each branch on each UNIX >>>> server. >>>> >>>> ==================================================================== >>>> 1. Notification of Build System >>>> ==================================================================== >>>> A post commit hook will notify the build system that a commit has >>>> occurred (I will need the assistance of systems guys set up this >>>> post-commit hook). >>>> >>>> A simple web service will be set up at the master to receive these >>>> notifications. >>>> >>>> The revision number will be entered into a queue and then a queue >>>> processing >>>> system will take it from there. >>>> >>>> The notification will be sent to the master and the other slaves will >>>> be notified after that. >>>> >>>> I am planning for the master to be the Debian server and the other >>>> servers >>>> will be the slaves. >>>> >>>> ===================================================================== >>>> 2. Processing items in the queue >>>> ===================================================================== >>>> The build will occur in very distinct phases/states. >>>> >>>> Each stage/state will be fine-tuned or broken down if needed in the >>>> future. >>>> >>>> We should be able to find out what state a particular job is at any >>>> give instant. >>>> >>>> Here are the states : >>>> >>>> 2A : The source code will be checked out of SVN. >>>> 2B : Then the buildconf command will be run. >>>> 2C : Then the configure script will be run will all the extensions >>>> enabled. >>>> 2D : Then make >>>> 2E : Then make install >>>> >>>> If any of the above steps fail, this particular job is put on HOLD and >>>> an email notification will be sent out. >>>> >>>> 2F : Then make test (or any other simpler alternative) >>>> >>>> 2G: The results will then be agregated an analyzed. This will be >>>> explained in 4 below >>>> >>>> However, we have to take care of the following cases/scenarios to >>>> avoid false alarms. >>>> >>>> 1. Removal of phpt scripts (a phpt script that was in the previous >>>> revision no longer exists) >>>> 2. Addition of new phpt scripts (a phpt script that never existed is >>>> now introduced) >>>> >>>> ===================================================================== >>>> 3. Handling Errors and Build-Failures >>>> ===================================================================== >>>> Whenever a particular build fails (steps 2A-E), human intervention may >>>> be needed because the reason could be a missing dependency such as >>>> libraries, tools etc. >>>> >>>> As for steps 2G, someone will have to fix the source code to make it >>>> PASS the test. >>>> >>>> If the problem is not in the source code, then the phpt script may >>>> have to be adjusted to make it pass the test. >>>> >>>> ===================================================================== >>>> 4. Intepretation of Build results >>>> ===================================================================== >>>> 2G1 : Existing phpt files. >>>> If the result changes from PASS to FAIL, a FAILURE notice will be sent >>>> out (RED) >>>> If the result changes from FAIL to PASS, a RESTORED notice will be >>>> sent out (GREEN) >>>> If the result remains at PASS, no notice will be sent out. >>>> If the result remains at FAIL, no notice will be sent out. >>>> >>>> 2G1 : Removed phpt files. >>>> A REMOVED notice will be sent out (ORANGE) >>>> >>>> 2G1 : Newly introduced phpt files. >>>> If the result is FAIL, a FAILURE notice will be sent out (RED) >>>> If the result is PASS, a PASS notice will be sent out (GREEN) >>>> >>>> ===================================================================== >>>> 4. Managing Results Data >>>> ===================================================================== >>>> The results will be stored in a MySQL database that will be archived >>>> from time to time. >>>> >>>> There will be a web interface to view the results of each test for >>>> each server and SVN URL >>>> >>>> ===================================================================== >>>> References : >>>> ===================================================================== >>>> >>>> >>>> http://wiki.php.net/internals/buildbot#operating_systems_available >>>> >>>> http://www.php.net/svn.php >>>> >>>> -- >>>> "Good Enough" is not good enough. >>>> To give anything less than your best is to sacrifice the gift. >>>> Quality First. Measure Twice. Cut Once. >>>> >>> >>> >>> >> >> -- >> David Soria Parra - Software Engineer >> >> >> Sun Microsystems GmbH >> Sonnenallee 1 >> 85551 Heimstetten >> Germany >> www.sun.de >> mailto: david.soriaparra@sun.com >> >> Amtsgericht Muenchen: HRB 161028 >> Geschäftsführer: Thomas Schroeder, Wolfgang Engels, Wolf Frenkel >> Vorsitzender des Aufsichtsrates: Martin Haering >> >> >> -- >> PHP Quality Assurance Mailing List >> <http://www.php.net/> >> To unsubscribe, visit: http://www.php.net/unsub.php >> >> > Hi David and Eric, The current state of the continuous integration project is that I am still developing the continuous integration system and it is not yet ready for initial testing/use. I will let everyone know as soon as it is available. David, I thought about slower machines as I was thinking about the queue-based master-slave setup too so I think I can modify my original design so that machines that are categorized as slow can adjust and configure how frequent they test and what revisions they test. The original idea was to test after every commit so that we can detect exactly when which commit/revision introduced the issue. So I think that if we skip revisions it might require additional investigation work to trace where the problem might have originated. I will like to incorporate Eric's recommendations and alter the original design so that faster computers can do all the revisions/commits and slower machines will do maybe just 4 revisions per day. Thanks everyone for your input. It is highly appreciated. -- "Good Enough" is not good enough. To give anything less than your best is to sacrifice the gift. Quality First. Measure Twice. Cut Once.