Planning, playbooks, and practice: building incident readiness

Gary Porcas, IR Consultant
March, 2021
10 mins read

What do great sports teams and incident response (IR) teams have in common? Firstly, they need multiple game plans or strategies which respond and adapt to their opponent’s tactics. Secondly, it takes repeated practice of a variety of scenarios to train their responses and enable performance under high-pressure conditions.

Planning and building readiness

IR teams rely on robust plans that contain a suite of playbooks. These playbooks guide responders’ actions in different compromise types ranging in severity.

Fig. 1. Flowchart demonstrating how playbooks for different compromise types fit together in an IR plan.  

Each playbook spans the complete IR lifecycle, from triage, first response, investigation and analysis, through to containment, remediation, and review. At each stage, the playbook contains decision points where responders decide their next steps. Some steps may require input from other teams and departments. Some steps may invoke other playbooks, or recommend contacting an external IR provider, depending on how the incident progresses.  

F-Secure Consulting IR playbook for containment

Fig. 2. The containment phase of an IR playbook for a host compromise. The diamonds represent decision points for responders’ next steps, while the rectangles represent actions to be taken. After successful containment, the IR team can proceed to the remediation phase. 

Testing your assumptions

To ensure business continuity and resilience, IR plans must cater to a range of compromise scenarios, but they must also remain adaptable to unforeseen complications. This isn’t limited only to advancements in attacker tradecraft. Changes to your IT estate, or obstacles that prevent timely access to people, systems, and information required to execute IR plans also present practical challenges.  

Take, for example, how the rapid shift to remote working in 2020 presented challenges to successful cyber incident response. If you don’t have physical access to your IT infrastructure and there is an attacker on your network, what options do you have to respond remotely? Consider the following questions: 

  • If an attacker compromises a server and you can’t physically switch it off, what do you do? 
  • If you can’t pull out network cables, can you take action against an attacker remotely to isolate them and delay their progress?  
  • If an attacker encrypts your systems, do you have backups available? Can you recover systems remotely? 
  • If an employee’s laptop is compromised, how will you access it? Will the laptop need to be shipped somewhere? Will incident responders and the employee need to come to the office? What kind of lead time will this add to the response? 
  • If an attacker compromises and is monitoring your internal communication channels, how will you coordinate the response?   

You can’t plan for every possible event and complication. For this reason, all IR plans are based, to some degree, on assumptions. These assumptions must be tested against real-world scenarios to validate the efficacy of the playbooks responders will use. The most beneficial time to do this is when IR plans are first being designed and developed. However, as illustrated by the remote working example, unexpected changes can render the steps outlined in a playbook obsolete. Research from 2020 suggests that 69% of businesses have changed their cyber security emergency response plans in light of the Covid-19 pandemic. Despite this, only 2% of organizations have run incident response scenarios related to the pandemic. 

This raises the question of how IR teams can identify unknown unknowns—the undiscovered and untested assumptions upon which their IR plan is built. It may help to bring in the external perspective of consultants who have first-hand experience of responding to a variety of cyber incidents. Bringing their knowledge of the practical challenges and omissions that hinder IR plans, they can help turn untested assumptions into working hypotheses. Crucially, hypotheses are a starting point for the IR plan, based on known facts and available evidence, that must be tested against real world scenarios. 

Validating your playbooks

Playbooks are both a guide for responders on how to approach an incident and a set of reliable instructions. However, when the underlying assumptions are not tested, playbooks can be limiting. An example described by one of our consultants begins with a user raising a helpdesk ticket saying, “my mouse is behaving funny.” When the team looked at the user’s endpoint and discovered malware, they followed their process: contain the malware on that machine and give the user a new one. Later, the same user reported the same problem. This triggered a lengthy investigation which discovered an attacker using a Remote Access Trojan (RAT) to control that employee’s machine outside of their work hours. In following their process, the incident responders did not investigate how the user’s job role related to why they specifically had been targeted. By not testing the assumptions on which their playbook was built, the team encountered a compromise scenario which they had not planned for.  

Tabletop exercises that simulate a security compromise from end to end enable IR teams to stress test their playbooks. Validating whether the actions in the playbook respond effectively to a range of incident types isn’t the only benefit of tabletop exercises. They also allow incident responders to familiarize themselves with a range of compromise scenarios and the actions in their playbooks. This will give them the confidence to act decisively during a crisis, both individually and as a team.  

Playbooks can be reviewed monthly using tabletop exercises, but we recommend that responders practice them as frequently as possible. These exercises don’t need to be onerous or time-consuming. Responders can practice every week or two, or even informally over lunch, to build recall and test their problem-solving skills within a complex scenario.  

Building resilience and ensuring business continuity

Certain types of security compromise require the involvement of non-security stakeholders such as PR, communications, operations, and the C-suite. When conducting tabletop exercises that simulate major incidents, the involvement of these stakeholders will ensure that, in such an event, their actions and decisions support business continuity.  

Our own consultants conducted a tabletop exercise with an energy supplier where the attacker first compromised the company’s IT infrastructure then progressed to their physical infrastructure. The scenario was designed so that the attack on their physical infrastructure would cause a major disruption to their service. After the exercise, the company’s internal IR team collaborated with other departments, building processes to ensure business continuity in the event of such a compromise. They adjusted their infrastructure so that the connection between their IT and their physical systems was reduced to a single cable. They then built processes to make sure there would be no disruption in service for their customers if the cable was unplugged. More importantly, the CISO gained authorization from the relevant stakeholders to sever this connection should their physical infrastructure be threatened by a malicious attacker. They also devised a recovery plan for the devices controlling their physical infrastructure. By copying a golden image from their server, the IR team could get new hosts up and running in a matter of hours.  

No plan survives first contact with the enemy. This adage doesn’t mean planning is futile, but rather that no single plan can offer certainty when you encounter a malicious attacker. Like the sports team who require multiple game plans, organizations defending themselves against cyber attacks require a range of options to remain adaptable yet resilient. 

A successful response to a cyber incident relies equally on validated playbooks, and the skills, knowledge, and practice of the team executing them. Where the IR plan dovetails with disaster recovery and business continuity, a successful response relies also on careful preparation and collaboration with other stakeholders. Test your assumptions, practice your playbooks, and continually build the lessons learned into your IR plan so that it becomes a living document. Doing so will help manage the residual risk of unforeseen consequences and make the organization more resilient to security compromises.  

Accreditations & Certificates

F-Secure Consulting (F-Secure Cyber Security (Pty) Ltd) is a level 4 contributor to B-BBEE with a procurement recognition level of 100%. Learn more and download our B-BBEE certificate. Click here to read the press release.

Follow us
@fsecure_consult F-Secure-Consulting f-secure-foundry fsecurelabs