Anti-Malware Testing Standards Organization (AMTSO), which F-Secure is a member of, had a meeting in Helsinki in May. During that meeting AMTSO members approved two new guidelines to be published.



The first new guideline is for "Whole Product Testing." The introduction of whole product testing is a very important development. It basically means that instead of testing each of the features of a product separately and trying to deduct the real-life protection provided by the product from that (sum-of-parts testing), the whole product is tested against real threats. Whole product testing will bring testing closer to reality and as such will guide development of security software to a direction that truly benefits users.

We at F-Secure are strong believers in defense in depth and as such welcome "whole product" approaches. Most users of security products do not really care which feature in their security suite protects them as long as they are kept safe. We have several layers of protection in our product and so does everyone else. Measuring each layer separately in a vacuum is just not the right way to evaluate the protection level provided by a product.

As the readers of our blog certainly know, the web is the number one infection vector today. A very typical infection scenario is SEO (Search Engine Optimization) poisoning: Criminals have tricked Google into listing their site very high in search results when the user searches e.g. for a current event. In a scenario like this F-Secure has three layers of defense in place (see image).



A "whole product" approach for testing protection against a threat like this could go like so:

   1) Take a URL that links to a drive-by-download exploit or malware download

   2) Browse to that URL with a web browser imitating a normal user

   3) See what happens. Does the malware infect the system or not?


One of the fundamental principles of AMTSO is that "testing must not endanger the public". So, a tester that conducts a test like the one above must take the necessary precautions e.g. make sure his network infrastructure prevents malware from attacking any outside systems.

The second new guideline released is about performance testing. It talks about scanning speed and resource usage. It is not just about "whole product performance testing" yet, so it is somewhat focused on testing individual aspects of performance. It gives sound advice on how performance aspects of security products can be evaluated. It especially highlights that the performance tests run should be relevant to the use-case in question. As an example, typically it does not make sense to test scanning speed by scanning infected files since most files a normal user would scan are clean. Also, while home user tests might focus on performance effects on computer games or media players, an enterprise file server focused test might concentrate more on on-demand scan performance.