We have been very busy recently (aren’t we always?) because we are preparing the final release of Linux Security 7. Even though we try our best to be agile and use Scrum, things still seem to speed up towards the end when every string is tied and every surface polished. We are excited about this release. Linux Security 7 has a number of new features and fixes we are sure you will like.

Last week I wrote about the importance of one command build. Among other benefits, it makes it easy to automate the build. And when you have an automated build, you can run the build as often as you want.

Why would you want to build often? Let me tell you…

Five years ago I worked in a team where we had a weekly build on Thursday afternoon. Every developer worked on their own module from Monday to Thursday morning and then did cvs commit. Then the Build Master made the build and invariably it failed to compile. Then the Build Master figured which developer to blame for the compilation failure and the poor developer tried to frantically make his code co-exist with all the changes others had done. When the developer committed his fix, the Build Master tried to build again. Lather, rinse, repeat.

When the Build Master finally got a build that compiled, he tried to install it. If it failed to install, we digged into it to find out why, fixed it and started over.

Sometimes it took until Friday afternoon before we had a build good enough for Quality Engineers to start testing. Needless to say, you could not commit anything during building or you could risk breaking the build.

This was a very stressful time! I really began to hate Thursdays.

When the next project started, I wanted the team to start building every night. People were generally against this saying it made no sense to build every night because the Quality Engineers could only test one build per week. Eventually I got permission to make an automated build script and make builds every night. (We agreed the Quality Engineers will test the Thursday build and pretend the other builds do not exist.)

It was much better! The nightly build revealed build problems earlier, every night really. The team no longer had to wait until Thursday afternoon to find out their source tree does not compile. Every morning the developers had email telling them if the build failed and it was pretty easy to see who should fix what. I was not stressed any more.

The Linux team has had automated nightly (well, daily, because we build at 4 pm) build from the start and we also do automated hourly builds.

An hourly build is even better than a daily build. You find out problems with your source every hour! This means you can fix them much sooner. And when you fix problems as early as possible, your daily build will be in much better shape. In fact, I don’t even remember the last time we had a broken daily build.

Why not take this even further? Why not build every 15 minutes? On every commit?

Currently we only build hourly because we test every hourly build. The hourly test is fairly comprehensive, it includes installation, basic functionality, some detection tests, even some WebUI tests and an uninstall at the end. All this testing takes some 40 minutes.

After Linux Security 7 is released, we plan to spread the test on multiple machines to make it faster. Hopefully we can then start building+testing every 15 minutes or less.

The holy grail of Continuous Integration would be to get immediate feedback when you do a bad commit. I’m not sure how we get to “immediate” but we keep on improving and that is what counts.

For a smallish team, communication and teamwork is usually smoothest when sitting in the same room. But, as the team grows, every person adds to the general noise and fuss levels in the room. People don’t just bring the noises they themselves make into the room. Every person also brings in more phone calls, visitors and spontaneous discussions.

We currently have 11 people sitting in the same room and I have to admit that at times our working environment is pretty hectic. We don’t get a lot of phone traffic, but it’s quite common to have a few conversations going on simultaneously accompanied by sound effects that state our latest hourly automatic test results. Luckily most of the team members are not easily disturbed, and don’t seem to be having concentration problems too frequently. The issue does come up at times though, and there has been one major “innovation” that has helped us a lot - Silent Hour.

Our daily scrum meeting is at 11.00 every day. After that there usually are some more discussions going on for a while, and then we go for lunch. Later, in the early afternoon, generally the most productive time of the day starts. To make sure it is used to the fullest, we have our Silent Hour (which actually lasts for 2 hours). We ourselves shut up (pair work with low-volume discussion is ok) and move any conversations to our IRC channel or out of the room, into the kitchen or a meeting room. A STOP sign on the door stops wannabe-visitors and asks them to lure whoever they want to talk to out of the room. Phones go silent as well.

We’ve had Silent Hours since last summer. At first we were afraid that our visitors might not appreciate it, but generally there have been no issues. Of course, every now and then someone just marches into the room to state their business and needs to be guided out, but surprisingly there have been no complaints over that. Actually, one co-worker thanked us for having the Silent Hour info on the door because it implicitly says that it’s OK to visit us outside the Silent Hour.

Sometimes we have forgotten to be silent ourselves. Still, the value of Silent Hour is verified by the fact that it hasn’t been forgotten for more than one day at a time.

Last week I wrote how important it is you can set up your build environment by running just one command.

Well, perhaps it is a worthy target to aim for, even if you miss it slightly. I showed you our build environment set up:

$ wget -O- http://server/1cmdenv.sh | sh
$ cd universe
$ make client-server-5.5x


Strictly speaking, that’s 3 commands (wget, cd, make). I’ll argue that cd is not really a “command”, as such, and since we have so many source trees you need 2 commands: 1 to check out the “tree-of-trees” or “universe” as we call it, and 1 to check out the source tree you want to build. I would say that is as close to one command build environment setup as we can get.

The other alternative would be to create separate setup commands for each source tree. Then where do you store these commands? Probably in the version control. But then you need another command to check them out. So, 2 commands is reasonably close to the ultimate goal.

Now to the subject of this post: The one command build.

We have chosen the one command to be: make. Running just “make” with no parameters performs a full build. With full build I mean it produces the installation package that is shipped on cdroms and uploaded to web sites.

Whatever the command is, it is important it does everything with no manual tweaking.

If there are manual steps in your build procedure, someone, somewhere, sometime when you least expect it, will get them wrong. Script it. All the way.

Even when your build is just one command, someone will think he is in a hurry and he doesn’t have time to run the one command. He will try to cut corners. He will think: “I already have module 1 built on my workstation. I’ll use that and build modules 2 and 3 by hand and execute the packaging command from the temp directory. I’ll save 10 minutes.” Beware! That is the way dragons lie.

What’s wrong with some shortcuts? Nothing if you can be absolutely sure you know what you are doing. Can you? Do you trust yourself? You do remember what causes most accidents in nuclear power plants? Human error.

4 months ago one man at the waterworks of Nokia, Finland (the city, not the phone maker) accidentally left one valve open and as a result contaminated the drinking water of tens of thousands of people. It took months to clean up the water pipes.

Long, long time ago I attended a lecture on fault tolerant systems. The lecturer said: “If you only remember one thing about this lecture, let it be this: The easiest way to build a fault tolerant system is to place the system in a very strong cabinet, lock the door and lose the key.”

In other words, eliminate the human factor.

When we run “make” in our source tree, it will do at least these things:

• generate dependencies
• export header files and string tables from a MIB file (contains all the settings and alerts of a software product in the F-Secure Policy Manager system)
• run a Perl script to generate code from the headers exported in previous step
• run another Perl script to verify string tables above match the localized string tables
• compile and install the unit test framework
• compile and run unit tests
• run a few static analyzers
• compile everything that is needed
• create RPM and DEB packages from the compiled code
• create a self-extracting installer from the RPM packages and an APT repository from the DEB packages
• export system tests from the source tree to be executed on the build by an automatic test system
• copy all the stuff that was generated to a separate file server

I’m sure I forgot a few steps in the above list. There’s too many to remember. The beauty of it is: I don’t have to. The one build command does it all for me.

When the build is just one command, it is easy to automate it. When the build just happens automatically, it further removes any possibility to screw it up. Create a cron job to make the build every night.

When you have a cron job making the build every night, why not make it every hour? Or every 15 minutes? When you have that, why would you bother making a build by hand ever again if a cron job is going to do it anyway in a few minutes. And it will do it better than you. Always the same way.

That sounds like Continuous Integration. Maybe I’ll write about that next week.

Just to make sure the message goes out: as mentioned by our colleagues in the lab, this week we released an important security advisory “FSC-2008-2″, about potential vulnerabilities affecting components common for many of our products, including Linux products. If you’re running one of our Linux products, chances are that you are affected and you should upgrade to the latest build immediately. Read the advisory for more information, or get more details from CERT-FI.

To get e-mail notifications about maintenance releases, subscribe to our maintenance release notification mailing list.

We are going to write a series of posts on good practices concerning software development, configuration management, building, testing and methodologies. The stuff in these posts will not be vague, abstract ideas but real lessons learned by veterans in the trenches of the battlefield.

This is the first post in the series, so let’s start from the beginning.

Everyone of us has been in this situation: You are the new guy in the team. You have just set up your computer. Email and web browser works. You are eager and ready to start coding!

What is the first thing you must do to start coding? You must install some tools and libraries. You must find out where to get the source code. Which version control tool do they use here? Where is the repository? Who do I talk to get access?

Do you remember how long it took before you were able to build the source code you were supposed to be working on? And build it in such a way that it really resulted in something that installs and works?

It probably took you anywhere from a few hours to a few days (and I have read about horror stories where it took weeks.)

Why?

Because the old hands in the team already have their own build environments set up and have had them for years. Their workstation has all the tools, all the libraries and all the little undocumented tweaks that are needed to make the perfect build. The build environment has not been set up, it has more like grown or evolved. Documentation, if any, is probably outdated.

The only way to avoid this mess is to make it possible to set up a complete build environment by running one (1) command in a clean workstation.

Why one command? If there are two commands, you might not remember them both or you might run them in the wrong order. If there are more commands, you definitely will not remember all of them.

And why would you bother with anything more than 1 command? If you currently need to run 2 commands, make a script that runs them. Now you have only one command. You have just automated the whole set up process and reduced the number of things you need to remember about it by 50%!

What this command does is entirely up to you. It can install software from network drives, copy files, check out source code, whatever is needed to turn a freshly installed Windows or Linux or Mac into something you could replace your build server with.

If you have such a command, it is probably a script. You do not need documentation, just read the script. If there is some obscure stuff in the script, write a comment into the script.

Test the script. Get your boss to buy you a new, shiny, fast workstation and use the script to set up your build environment on it. (If your boss refuses, set up a fresh virtual machine or something.)

And best of all, if one day you see smoke coming out of your build server, you know how to make a new server in no time at all.

We in the Linux team have a script that uses apt-get to install bazaar, configures it, then checks out an Arch project that has a Makefile. This Makefile is used to check out the various source trees we work with. The Makefile also knows which tools you need for which source tree and will install the tools before checking out the source.

This is how it would look if you wanted to make a build in a clean computer:

# One command build environment setup
$ wget -O- http://server/1cmdenv.sh | sh
$ cd universe
# Check out one source tree and install the tools it needs
$ make client-server-5.5x
$ cd client-server-5.5x
$ make testbuild
# Now install and test the build in ./release directory


Of course, it may not be a good idea to depend on tools and libraries that are part of your OS. When you upgrade the OS, you sometimes get new versions of them that are not compatible with the way you use them. But that is a topic for another post.

We have just released a new version of F-Secure Linux Client & Server Security. This 5.54 is mainly a bug fix release, including a new Automatic Update Agent and making the Web User Interface work properly with Microsoft Internet Explorer 7. Please check the release notes for more information on what has changed. As always, we recommend upgrading to the latest version even if any of the fixed issues do not seem to affect your environment. Please feel free to ask if you have any questions regarding this upgrade, and do drop us a note telling how the upgrade went :) The email address is at the bottom of this page.

The product packages and release notes are available from our Webclub:
F-Secure Linux Client Security 5.54
F-Secure Linux Server Security 5.54

As you may have read between the lines of our blog posts so far, we’re an agile team. We use Scrum to do our thing. In practice this means that instead of making rigid plans for a long time ahead and then following them, we split the work into iterations and try to adapt quickly as we learn more and the customer and market needs change.

This is how it works. We split our release projects into (usually) one month sprints. In the beginning of each sprint, we select the most important things to work on next and do detailed planning only for them. As you have probably noticed, we also strive to have some new features ready and released every sprint. The main idea here is to get feedback for the things we do as quickly as possible so that we can learn and adapt accordingly.

The point of this post is just to make sure you know we really (yes, rly :)) appreciate every little piece of the feedback you have sent us so far and hope you keep it up!

Linux Security 7 Beta 3 is out now and again we have several new supported platforms. There are now 28 supported Linux distribution versions listed in the release notes and actually we have 36 of them active in nightly automated tests. Yes, there are problems for example on Fedora 7 and TurboLinux 11 that haven’t been solved yet, but they will most probably be supported when the final release arrives. I took a quick look at competitors web sites and it looks like we have currently the widest platform support in the Linux antivirus industry.

This is of course enabled by automation. If you want to see how our test automation system has evolved, see my test automation presentation in Google Test Automation Conference last fall in New York.

A test module called moosetest mentioned in the presentation has actually enabled us to triple the amount of test cases run per platform after the presentation took place. We can now re-initialize, reboot, crash and torture the test computers to our heart’s content. Also for the first time we have full coverage for the upgrade-from-previous-versions test scenarios. Also the test system now runs fully automated performance tests on real hardware in addition to normal testing performed on virtual machines. “fair simulation” philosophy for the moosetest was adopted from the Saab version of moose test. See the wikipedia article on moose test.

I can already say that this release will be the best quality we have ever released. Now go get the beta!

Happy new year everyone! Today we are introducing the third beta release of Linux Security 7.00. This is mainly a bug fix release and does not contain any completely new features. We have also extended our list of supported platforms with 32-bit versions of Asianux 2.0, Asianux 3.0, and Miracle Linux 3.0.
Some other new supported platforms are also on their way, including Fedora Core 7 and Turbolinux 11, but could not quite make it into this release.
Please find the full product package here: f-secure-linux-security-7.00.70103.tgz (MD5,
SHA1).
Be sure to read the release notes before installing. As usual, any comments and feedback you might have is greatly appreciated, so please send them to the address at the bottom of this page!

Even though it’s just barely 10 days since we announced our last 7.00 beta release, we’re today presenting another one.

This time around we’ve got a couple of new features, although most of the work has gone into overall quality and stability improvements, much like with the last release. The manual scanning wizard has been redesigned to show a progress bar during scanning, and the Summary screen has gotten a new row showing the current Integrity Checking status, which also gives the user an opportunity to initialize the file system baseline and enable the rootkit protection through a wizard. The on-access scanner component has been removed from the `fschooser‘ program - instead the user can disable the Real-time Scanning feature completely by setting “Virus Protection” and “Integrity Protection” to “Disabled” in the web user interface.

Here are the usual links - full package: f-secure-linux-security-7.00.64805.tgz (MD5, SHA1). As always, please read the release notes and send us feedback to the address at the bottom of this page.