Nicholas Evans, Research Analyst
10 mins read
Building a secure software development lifecycle is a universal good. It leads to better products. But without other changes being made, it isn’t going to fix security problems in the development environment. For that, we need to examine how security operations are run.
I only came to this realization very recently. Like most people, I had assumed that shifting security to the left was a process that would take more time. Where it wasn’t working, I would have told you that more effort was required to allow a security culture to permeate software development. But then I started speaking to security professionals from across the industries in which F-Secure works, and something became quickly apparent. Even though developers are writing much more secure code, the security of development environments doesn’t seem to be getting better. And that’s despite concerted effort.
In fact, my own industry research showed that development environments are a top security worry for CISOs attempting to secure their cloud estates, irrespective of how long their organization has been shifting security to the left.
We’ve all been trying to shift left for close to half a decade. If, in that time, cyber mature financial institutions haven’t managed to do it, then I’d confidently say that nobody is going to get there soon.
Responders on the front lines have known this for a while. Joani Green—who leads F-Secure’s UK incident response team—says, "misconfigurations caused by decentralized DevOps teams have been the primary reason for the cloud breaches we've investigated in the past year or two. Primarily this is about DevOps pipelines containing secret keys, organizations not having control over secrets, and also logging and configuration tracking."
So what can explain this? And how can we make things better?
I’ve been thinking about these questions a lot recently. In so doing I’ve realized that there’s a larger story here. This is a story about how our attack surfaces changed but our approach to securing them lagged, and of how we all thought that cultural changes could fix organizational problems.
This is a story about how we all thought that cultural changes could fix organizational problems.
To understand why we are in this situation we have to look at the history of the corporate security function. Information security began life as a segmented part of the business because it was seen as a specialist function that protected IT (already a specialist part of the business). Information security was—and usually continues to be—an outgrowth of IT.
One of F-Secure’s experts in the area, Matthew Pendlebury, summarizes the situation: “you typically end up with security operations being associated with IT operations, because that’s where it came from when it was all about firewalls and pen tests.” And of course, this model made perfect sense when infosec was all about those firewalls and pen tests.
But then, about a decade ago, things started to change in a way that we are all familiar with. Organizations began to move to the cloud. Processes became scalable in a way that had never been the case before. Development work got really fast.
As such, software developers increasingly became responsible for creating a large percentage of the vulnerabilities that make up a typical enterprise organizations’ attack surface.
“You’ve got this really big target that is in the cloud—and outside of your building’s perimeter – and it is under the control of your development team rather than under the control of your IT team,” Matthew explains. And he should know. Matthew worked for 17 years as both a developer and running development teams delivering secure code in secure environments, before joining F-Secure as a consultant specializing in inserting the ‘Sec’ into DevSecOps. Now he runs large parts of our internal security operations. When you ask—who secures the security specialists? — Matthew’s name is the answer.
“You’ve got this really big target that is in the cloud—and outside of your building’s perimeter – and it is under the control of your development team rather than under the control of your IT team.”
Matthew’s words really point to the nub of the problem. Infosec professionals are largely still aligned with IT. They sit on the perimeter of networks, over which they exercise remarkable control. But on-prem networks are the source of yesterday’s problems. The cloud has blown apart that idea of the perimeter and made most of the old techniques for defending it obsolete.
In summary, Infosec teams are no longer organized in a way that makes them best placed to deal with emerging vulnerabilities. It’s no wonder that many individuals working in Security Operations Centers, as well as security focused leadership teams, see developers as unruly liabilities. One infosec leader told me that “DevOps is still the wild west.” This judgment might not be particularly fair on the developers, but can anyone really blame him for thinking this way, when developers remain so completely beyond his control?
A 2001 blog by Larry Smith is often credited as the first statement about shifting security to the left. Smith was talking about testing in a narrow sense, and he was also writing over a decade before most organizations were even thinking about the cloud. Nonetheless, in the last decade, his basic insight has been developed into the dominant means for organizations to try and control the security problems that arise with decentralized development.
In this decade, a huge amount of progress has been made in securing development processes. Developers who think about security as a core feature of their products write code that is more stable, robust, and resilient. They have clearly demonstrated that there is a business cases to be made for adopting a DevSecOps process that goes beyond just ‘securing the product’. But then, there are also things that simply haven’t been addressed by most organizations going down this path.
Crucially, many organizations still don’t factor the cost of security into individual business functions. The result, says consulting director Jesper Gerved, “is that security doesn’t get implemented because it’s not on organizations’ priority lists.” If an organization wants to distribute security operations out into its business functions, it needs to turn security into a measurable deliverable for employees, equal to the other goals it pursues. Developers are under intense pressure. A recent study (with an admittedly tiny sample size of 260) found that 83% report feeling burned out. These are people who are already being pushed to breaking point. If security stops them meeting targets that their organizations value, they will find ways around security. Matthew puts this all in simple terms: “developers have a split goal of needing to do security stuff but having their backs up against the wall to cut code and deliver features.”
What often happens is that developers get asked to secure their projects, even though financial responsibility for that security lies with a separate operational team. Matthew (again) is very clear: “you need to be able to solve that divergence. To figure out what security is worth for your organization.” Ultimately, he explains, “the question is who’s going to pay for it.”
This is a problem that our consultants working in the field are increasingly coming up against. Antti Vähä-Sipilä is one of the pioneers of securing the systems development lifecycle (SDLC) in his native Finland, with big plans to revolutionize how organizations think about their security architecture. But Antti has found that simply focusing on improving processes is no longer always enough to achieve the changes needed to make his clients secure. “The root causes of these problems run much deeper than an SDLC gap analysis can expose,” he says, referring to the kind of assessment that he usually carries out when starting work with a new client. Gaps in SDLC processes get spotted, but fixing them doesn’t lead to the necessary improvements, and the reason for this is simple: many companies now have organizational structures that are “unhelpful for getting security right."
…many companies now have organizational structures that are “unhelpful for getting security right."
Antti thus sees organizational change as being crucial to overcoming these challenges. His actual recommendations—which have to do with restructuring entire product management concepts—fall outside the scope of the reports that he is usually asked to write. Often, what is needed are a series of tough but honest discussions with those at the very top.
Building a security culture among developers is a necessary and important goal, but it doesn’t solve the problems that I’ve been discussing here. For that, a change in organizational structure is needed, preferably one that takes infosec away from its association with IT, and closer to the business functions of an organization.
I think it’s only a matter of time before we start to see this reflected in the structure of security operations, and in the kinds of security jobs on offer. We are already seeing this with isolated organizations. At least one CISO with whom I spoke is experimenting with a radically new model of having an entirely ‘virtual’ operations team. All the members of this team have a main role in another function and then work part time for the CISO. It’s a start, but it still suffers from the problem of business goals dominating those of security, and it tends to put individuals in an awkward personal position of having to deliver features and the security controls that might hinder those features.
What I can say with certainty is that things are going to change, and having spoken to a number of experts on this topic, I’m willing to make three strong predictions.
Prediction 1: Business functions will start paying for their own security
The lifetime cost of making software secure will routinely be built into the budgets of individual projects. This will help resolve tensions between business goals and security goals. Product managers will learn that remediation work on poorly-built applications can cost many times the amount required to build things securely at the beginning. Once managers start factoring in the cost of incident response, the financial case for early security intervention will become ever clearer.
Prediction 2: The CISO role will become more independent and more strategic
CISOs will increasingly become independent of IT departments. They will report directly to CEOs, and they will have more flexibility to set the direction of strategy. This should lead to development of a clear a security strategy, and less mopping-up in the wake of others.
The flip side of this change is that CISOs will be less responsible for day-to-day security operations. Jesper is one supporter of this view: he believes that the future CISO will provide direction to the business and monitor progress without necessarily being responsible for implementing security operations. Consequently, security operations will be distributed down into individual business units, to avoid backlogs, and ensure that those who work with specific technologies are also securing them.
Future CISOs will nonetheless continue to coordinate services that work best when centralized. Jesper sees the organization’s red team as one such “service function to the business.” Incident Response is another such service, because it requires a set of skills which “you simply cannot maintain down in each and every business unit.”
Prediction 3: A new middle layer will emerge
We’ve already seen that too much centralization can be bad. But when it comes to security, so is too much distribution. Thomas Wearing—who works with Jesper in security management—points out that a complete decentralization of security would lead to radically different risk appetites being adopted across products. There is also the fact that you need somebody with working knowledge of the product to help translate the top-down policy of the newly transformed CISO office.
Most of the experts I’ve spoken to therefore believe that a mid-level role will need to be defined and adopted in the very near future. “Sometimes there are these pseudo-roles, but they are always a bit fluffy and it’s not very well defined,” explains Thomas. “I think you are going to see a role created which is more about coordination and consolidation. But I have yet to meet someone who has that role properly.”
Matthew agrees. “You need a mid-level, which is somebody who is closer to the product, who can take responsibility more locally,” he says. “I’m thinking here of the equivalent of an Engineering Manager for that product.” The idea would be somebody who can see across multiple teams, but also has a deep knowledge of the products being developed. Somebody who can get down into the nitty gritty. Above all, it would be somebody who could make security a business-critical goal for each product.
This organizational layer would take over many of the repetitive tasks that currently keep CISOs bogged down in detail, but crucially, it would also allow responsibility for security to become embedded in the work cycles of individual developers.
Like everybody always says, there’s no quick fix to securing a development environment, but part of the problem up until now has been that our focus has been too narrow. We’ve obsessed over changing how developers think, when that’s only part of the puzzle. We also need to change how we organize our security operations.