Stuff That Works part 4: Continuous Integration
We have been very busy recently (aren’t we always?) because we are preparing the final release of Linux Security 7. Even though we try our best to be agile and use Scrum, things still seem to speed up towards the end when every string is tied and every surface polished. We are excited about this release. Linux Security 7 has a number of new features and fixes we are sure you will like.
Last week I wrote about the importance of one command build. Among other benefits, it makes it easy to automate the build. And when you have an automated build, you can run the build as often as you want.
Why would you want to build often? Let me tell you…
Five years ago I worked in a team where we had a weekly build on Thursday afternoon. Every developer worked on their own module from Monday to Thursday morning and then did cvs commit. Then the Build Master made the build and invariably it failed to compile. Then the Build Master figured which developer to blame for the compilation failure and the poor developer tried to frantically make his code co-exist with all the changes others had done. When the developer committed his fix, the Build Master tried to build again. Lather, rinse, repeat.
When the Build Master finally got a build that compiled, he tried to install it. If it failed to install, we digged into it to find out why, fixed it and started over.
Sometimes it took until Friday afternoon before we had a build good enough for Quality Engineers to start testing. Needless to say, you could not commit anything during building or you could risk breaking the build.
This was a very stressful time! I really began to hate Thursdays.
When the next project started, I wanted the team to start building every night. People were generally against this saying it made no sense to build every night because the Quality Engineers could only test one build per week. Eventually I got permission to make an automated build script and make builds every night. (We agreed the Quality Engineers will test the Thursday build and pretend the other builds do not exist.)
It was much better! The nightly build revealed build problems earlier, every night really. The team no longer had to wait until Thursday afternoon to find out their source tree does not compile. Every morning the developers had email telling them if the build failed and it was pretty easy to see who should fix what. I was not stressed any more.
The Linux team has had automated nightly (well, daily, because we build at 4 pm) build from the start and we also do automated hourly builds.
An hourly build is even better than a daily build. You find out problems with your source every hour! This means you can fix them much sooner. And when you fix problems as early as possible, your daily build will be in much better shape. In fact, I don’t even remember the last time we had a broken daily build.
Why not take this even further? Why not build every 15 minutes? On every commit?
Currently we only build hourly because we test every hourly build. The hourly test is fairly comprehensive, it includes installation, basic functionality, some detection tests, even some WebUI tests and an uninstall at the end. All this testing takes some 40 minutes.
After Linux Security 7 is released, we plan to spread the test on multiple machines to make it faster. Hopefully we can then start building+testing every 15 minutes or less.
The holy grail of Continuous Integration would be to get immediate feedback when you do a bad commit. I’m not sure how we get to “immediate” but we keep on improving and that is what counts.
