For a while now we have been doing incremental work to push out new RescueCD with new features.

So what new do we have?
Here is a Quick Changelog

• New Security Platform
• New Knoppix
• USB stick improvements
  • Ability to speed up the process by storing malware definitions on USB stick if scan is done often
  • Ability to automate scanning or enable malware definition download behind a proxy
  • Ability to run personalized automated script for the system as part of the process
  • Ability to convert RescueCD into bootable USB stick

New Knoppix

We have taken the Knoppix 6.0.1 as the new base for the RescueCD. It has been stripped down somewhat to make download a bit smaller and initialization process is slightly altered to make it fairly simple to use. If you think some crucial tool is missing from the image please let us know and we consider including it later. Both emacs and vi are removed on purpose :)

New Security Platform

Security Platform has been updated to version 2.50. It should be faster now.

Extended usability of USB stick with the product

Using USB stick to store malware definitions

To use USB stick as malware definition storage initialize USB stick by creating directory fsecure on the stick and under that subdirectory rescuecd. Now when you boot the machine with RescueCD and USB stick is inserted into machine it will automatically use the stick to store definitions into. (there must be ~256MB of free space on the stick)

To automate the scan process with help of USB stick

Act as above and after you have booted up you can have a look at the config file /mnt/usbstick/fsecure/rescuecd/config. This file explains different options that can be useful to automate the process or enable virus definition downloads by setting a proxy.

If you wish to automate the scanning process just set the TIMEOUT=5 and every dialog will show only for 5 seconds and after that proceed to next step. Process will halt to last screen that shows scan result.

If you are behind a proxy you can set the http_proxy in the config file according explained specifications.

Running personalized script as part of the process

If you have initialized the USB stick you can create file /mnt/usbstick/fsecure/rescuecd/custom_script.sh

This script can contain your own customised actions that are taken before scan process. This is basically same thing as repair_script.sh used to be, but just renamed to have more generic meaning. You can for example use this script to backup all pictures or documents from your machine to the stick (if it is large enough) or have rsync backup files to remote machine over network.

Converting RescueCD into bootable USB stick (for advanced users)

One of the nice features that we managed to push to the CD is transmogrify script. This script can be used to change normal USB stick into bootable RescueCD stick. Word of warning thou the script will delete everything from target device. So be very careful as you use it to create your bootable USB stick.

To create a USB stick version of the RescueCD, please run the following command in ‘Alt-F2′ after booting the CD:
/opt/f-secure/transmogrify_cd_into_stick.sh [cdrom device] [usb stick device]

[cdrom device] usually is one of the following /dev/hda /dev/hdc /dev/sr0. You can verify the device by changing to different virtual console with ‘Alt-F2′ and typing df. The device on the very first line with /mnt-system is the device you wish to use as first argument to the script.

[usb stick device] is bit harder to figure out. If you have just booted the computer without any stick and have the one that you wish to use in your hand insert it in and check dmesg | grep -A2 usb-storage. Command should print out something like

usb-storage: waiting for device to settle before scanning
scsi 126:0:0:0: Direct-Access USB 2.0 USB Flash Drive 0.00 PQ: 0 ANSI: 2
sd 126:0:0:0: [sdd] 3948544 512-byte hardware sectors (2022 MB)

so in this case destination device would be /dev/sdd
Note: if you create USB stick after databases have been downloaded you will have the fresh virus description databases already on your next boot.

The package can be downloaded here.
Please have a look at the release notes before using the product. They are also available inside the zip package.

checksums for f-secure-rescue-cd-beta-3.10.22900.zip:
md5sum: bbef00aa8e0be2c6398cd1cdfba71470
sha1sum: 12a339b755323c170eb2b1c0e2ddcc88b1b0f0cb

And as before comments and suggestions are welcome trough email address displayed at the bottom of the page.

Many thanks to KNOPPIX team NTFS-3G developers and people in OSS scene.

Last year, when we were writing a series of posts titled ”Stuff that works” (part 1, part 2, part 3, part 4), we were using a home-grown set of shell scripts to automatically build our software. Initially the scripts were run from cron every night, then later every hour and even later the script was triggered for every commit.

The autobuilder script worked, but we were missing features like IRC notifications, triggering builds from a web page and IRC and statistics. We would also have liked to have a history of builds combined with the test results of every build.

We were already setting up a MySQL database for builds and test results when we discovered that Continuous Integration is actually a hot topic these days and there is plenty of software available to help you with it. We eventually decided on software called Hudson.

Hudson is open source software and it is under active development by a large and growing community.
Hudson is written in Java and packaged so it can be set up easily and quickly. Basically you download the hudson.war file and run it: java -jar hudson.war

That’s it! Then you point your browser to port 8080 on that host and you have a pretty Web UI for setting up, managing and monitoring of various build jobs.

In a nutshell a job consists of following steps:

Poll a version control repository for changes.

Check out the source to a workspace directory.

Execute a build script.

Archive build results from workspace (or build artifacts like Hudson calls them.)

(Optional) pick up test results from XML files in workspace.

Each build of each job has a nice web page showing if the build was successful, when it was run and how long it took. The build artifacts can be downloaded from the same page and you can also inspect the test results.
You can configure a job to trigger one or more other jobs. For example, when the job that builds F-Secure Mac Protection finishes successfully, we have set it to trigger 2 other jobs: Installtest and Smoketest.

A job does not have to build software. Hudson does not care what the build script does. The build script in Installtest downloads the last successful build artifact from the job that launched it and installs it on a Mac mini running on a side table in the corner of the team room. Then it makes couple of simple checks and uninstalls the software. The result of this run is recorded in an XML file in the JUnit format. When the build script exists successfully, Hudson reads the XML file and stores the test results.

Hudson executes the Installtest parallel with the Smoketest job. Smoketest takes much longer to run (about 7-8 minutes), which is why we have the Installtest to provide us fast feedback in case we break something. Smoketest is similar to Installtest, except it runs many more tests.

We have also a number of other jobs, which are triggered by the Smoketest: Fulltest (runs all the tests taking about an hour), Upgradetest (runs upgrade from previous version and all tests from fulltest) and Performancetest (runs a set of common user tasks, measuring the time it takes.)

As you see, it is a good practice to split long jobs into multiple smaller jobs that run quicker. The quicker a job runs, the faster you get to know if you broke something and the faster you can fix it.

Hudson also has a whole lot of plugins that can be used to enhance and extend the basic functionality. There are plugins for supporting most of the popular version control systems and plugins to analyze source code and create reports and lots more. (And there is a plugin for IRC too.)

If you are doing continuous integration you could do a lot worse than use Hudson to help you. I definitely recommend you check it out.

We have been very busy recently (aren’t we always?) because we are preparing the final release of Linux Security 7. Even though we try our best to be agile and use Scrum, things still seem to speed up towards the end when every string is tied and every surface polished. We are excited about this release. Linux Security 7 has a number of new features and fixes we are sure you will like.

Last week I wrote about the importance of one command build. Among other benefits, it makes it easy to automate the build. And when you have an automated build, you can run the build as often as you want.

Why would you want to build often? Let me tell you…

Five years ago I worked in a team where we had a weekly build on Thursday afternoon. Every developer worked on their own module from Monday to Thursday morning and then did cvs commit. Then the Build Master made the build and invariably it failed to compile. Then the Build Master figured which developer to blame for the compilation failure and the poor developer tried to frantically make his code co-exist with all the changes others had done. When the developer committed his fix, the Build Master tried to build again. Lather, rinse, repeat.

When the Build Master finally got a build that compiled, he tried to install it. If it failed to install, we digged into it to find out why, fixed it and started over.

Sometimes it took until Friday afternoon before we had a build good enough for Quality Engineers to start testing. Needless to say, you could not commit anything during building or you could risk breaking the build.

This was a very stressful time! I really began to hate Thursdays.

When the next project started, I wanted the team to start building every night. People were generally against this saying it made no sense to build every night because the Quality Engineers could only test one build per week. Eventually I got permission to make an automated build script and make builds every night. (We agreed the Quality Engineers will test the Thursday build and pretend the other builds do not exist.)

It was much better! The nightly build revealed build problems earlier, every night really. The team no longer had to wait until Thursday afternoon to find out their source tree does not compile. Every morning the developers had email telling them if the build failed and it was pretty easy to see who should fix what. I was not stressed any more.

The Linux team has had automated nightly (well, daily, because we build at 4 pm) build from the start and we also do automated hourly builds.

An hourly build is even better than a daily build. You find out problems with your source every hour! This means you can fix them much sooner. And when you fix problems as early as possible, your daily build will be in much better shape. In fact, I don’t even remember the last time we had a broken daily build.

Why not take this even further? Why not build every 15 minutes? On every commit?

Currently we only build hourly because we test every hourly build. The hourly test is fairly comprehensive, it includes installation, basic functionality, some detection tests, even some WebUI tests and an uninstall at the end. All this testing takes some 40 minutes.

After Linux Security 7 is released, we plan to spread the test on multiple machines to make it faster. Hopefully we can then start building+testing every 15 minutes or less.

The holy grail of Continuous Integration would be to get immediate feedback when you do a bad commit. I’m not sure how we get to “immediate” but we keep on improving and that is what counts.

For a smallish team, communication and teamwork is usually smoothest when sitting in the same room. But, as the team grows, every person adds to the general noise and fuss levels in the room. People don’t just bring the noises they themselves make into the room. Every person also brings in more phone calls, visitors and spontaneous discussions.

We currently have 11 people sitting in the same room and I have to admit that at times our working environment is pretty hectic. We don’t get a lot of phone traffic, but it’s quite common to have a few conversations going on simultaneously accompanied by sound effects that state our latest hourly automatic test results. Luckily most of the team members are not easily disturbed, and don’t seem to be having concentration problems too frequently. The issue does come up at times though, and there has been one major “innovation” that has helped us a lot - Silent Hour.

Our daily scrum meeting is at 11.00 every day. After that there usually are some more discussions going on for a while, and then we go for lunch. Later, in the early afternoon, generally the most productive time of the day starts. To make sure it is used to the fullest, we have our Silent Hour (which actually lasts for 2 hours). We ourselves shut up (pair work with low-volume discussion is ok) and move any conversations to our IRC channel or out of the room, into the kitchen or a meeting room. A STOP sign on the door stops wannabe-visitors and asks them to lure whoever they want to talk to out of the room. Phones go silent as well.

We’ve had Silent Hours since last summer. At first we were afraid that our visitors might not appreciate it, but generally there have been no issues. Of course, every now and then someone just marches into the room to state their business and needs to be guided out, but surprisingly there have been no complaints over that. Actually, one co-worker thanked us for having the Silent Hour info on the door because it implicitly says that it’s OK to visit us outside the Silent Hour.

Sometimes we have forgotten to be silent ourselves. Still, the value of Silent Hour is verified by the fact that it hasn’t been forgotten for more than one day at a time.

Last week I wrote how important it is you can set up your build environment by running just one command.

Well, perhaps it is a worthy target to aim for, even if you miss it slightly. I showed you our build environment set up:

$ wget -O- http://server/1cmdenv.sh | sh
$ cd universe
$ make client-server-5.5x


Strictly speaking, that’s 3 commands (wget, cd, make). I’ll argue that cd is not really a “command”, as such, and since we have so many source trees you need 2 commands: 1 to check out the “tree-of-trees” or “universe” as we call it, and 1 to check out the source tree you want to build. I would say that is as close to one command build environment setup as we can get.

The other alternative would be to create separate setup commands for each source tree. Then where do you store these commands? Probably in the version control. But then you need another command to check them out. So, 2 commands is reasonably close to the ultimate goal.

Now to the subject of this post: The one command build.

We have chosen the one command to be: make. Running just “make” with no parameters performs a full build. With full build I mean it produces the installation package that is shipped on cdroms and uploaded to web sites.

Whatever the command is, it is important it does everything with no manual tweaking.

If there are manual steps in your build procedure, someone, somewhere, sometime when you least expect it, will get them wrong. Script it. All the way.

Even when your build is just one command, someone will think he is in a hurry and he doesn’t have time to run the one command. He will try to cut corners. He will think: “I already have module 1 built on my workstation. I’ll use that and build modules 2 and 3 by hand and execute the packaging command from the temp directory. I’ll save 10 minutes.” Beware! That is the way dragons lie.

What’s wrong with some shortcuts? Nothing if you can be absolutely sure you know what you are doing. Can you? Do you trust yourself? You do remember what causes most accidents in nuclear power plants? Human error.

4 months ago one man at the waterworks of Nokia, Finland (the city, not the phone maker) accidentally left one valve open and as a result contaminated the drinking water of tens of thousands of people. It took months to clean up the water pipes.

Long, long time ago I attended a lecture on fault tolerant systems. The lecturer said: “If you only remember one thing about this lecture, let it be this: The easiest way to build a fault tolerant system is to place the system in a very strong cabinet, lock the door and lose the key.”

In other words, eliminate the human factor.

When we run “make” in our source tree, it will do at least these things:

• generate dependencies
• export header files and string tables from a MIB file (contains all the settings and alerts of a software product in the F-Secure Policy Manager system)
• run a Perl script to generate code from the headers exported in previous step
• run another Perl script to verify string tables above match the localized string tables
• compile and install the unit test framework
• compile and run unit tests
• run a few static analyzers
• compile everything that is needed
• create RPM and DEB packages from the compiled code
• create a self-extracting installer from the RPM packages and an APT repository from the DEB packages
• export system tests from the source tree to be executed on the build by an automatic test system
• copy all the stuff that was generated to a separate file server

I’m sure I forgot a few steps in the above list. There’s too many to remember. The beauty of it is: I don’t have to. The one build command does it all for me.

When the build is just one command, it is easy to automate it. When the build just happens automatically, it further removes any possibility to screw it up. Create a cron job to make the build every night.

When you have a cron job making the build every night, why not make it every hour? Or every 15 minutes? When you have that, why would you bother making a build by hand ever again if a cron job is going to do it anyway in a few minutes. And it will do it better than you. Always the same way.

That sounds like Continuous Integration. Maybe I’ll write about that next week.

We are going to write a series of posts on good practices concerning software development, configuration management, building, testing and methodologies. The stuff in these posts will not be vague, abstract ideas but real lessons learned by veterans in the trenches of the battlefield.

This is the first post in the series, so let’s start from the beginning.

Everyone of us has been in this situation: You are the new guy in the team. You have just set up your computer. Email and web browser works. You are eager and ready to start coding!

What is the first thing you must do to start coding? You must install some tools and libraries. You must find out where to get the source code. Which version control tool do they use here? Where is the repository? Who do I talk to get access?

Do you remember how long it took before you were able to build the source code you were supposed to be working on? And build it in such a way that it really resulted in something that installs and works?

It probably took you anywhere from a few hours to a few days (and I have read about horror stories where it took weeks.)

Why?

Because the old hands in the team already have their own build environments set up and have had them for years. Their workstation has all the tools, all the libraries and all the little undocumented tweaks that are needed to make the perfect build. The build environment has not been set up, it has more like grown or evolved. Documentation, if any, is probably outdated.

The only way to avoid this mess is to make it possible to set up a complete build environment by running one (1) command in a clean workstation.

Why one command? If there are two commands, you might not remember them both or you might run them in the wrong order. If there are more commands, you definitely will not remember all of them.

And why would you bother with anything more than 1 command? If you currently need to run 2 commands, make a script that runs them. Now you have only one command. You have just automated the whole set up process and reduced the number of things you need to remember about it by 50%!

What this command does is entirely up to you. It can install software from network drives, copy files, check out source code, whatever is needed to turn a freshly installed Windows or Linux or Mac into something you could replace your build server with.

If you have such a command, it is probably a script. You do not need documentation, just read the script. If there is some obscure stuff in the script, write a comment into the script.

Test the script. Get your boss to buy you a new, shiny, fast workstation and use the script to set up your build environment on it. (If your boss refuses, set up a fresh virtual machine or something.)

And best of all, if one day you see smoke coming out of your build server, you know how to make a new server in no time at all.

We in the Linux team have a script that uses apt-get to install bazaar, configures it, then checks out an Arch project that has a Makefile. This Makefile is used to check out the various source trees we work with. The Makefile also knows which tools you need for which source tree and will install the tools before checking out the source.

This is how it would look if you wanted to make a build in a clean computer:

# One command build environment setup
$ wget -O- http://server/1cmdenv.sh | sh
$ cd universe
# Check out one source tree and install the tools it needs
$ make client-server-5.5x
$ cd client-server-5.5x
$ make testbuild
# Now install and test the build in ./release directory


Of course, it may not be a good idea to depend on tools and libraries that are part of your OS. When you upgrade the OS, you sometimes get new versions of them that are not compatible with the way you use them. But that is a topic for another post.