Saved by the log: Building IR support into software design

Thomas Wearing, Security Consultant and Jordan LaRose, Incident Response Consultant

4 minute read

A year ago, a group of our consultants put together a proposal for improving the development of attack-aware applications through app-level purple teaming. In this follow-up article, veteran Incident Responder Jordan LaRose, and Security and Risk Management Consultant Thomas Wearing join forces to explore how the design of attack-aware applications might be refined even further by anticipating the needs of incident response.

Here’s a statistic. Recent analysis by Forrester suggests that 39% of all external attacks begin with a web application exploit. In our industry, you can find a number to support just about anything (provided you’re willing to look hard enough). But 39% is high, even by the inflated standards of infosec.

So what’s going on? Unfortunately, some web apps are just full of vulnerabilities, and that is obvious to external attackers. But it’s not just the obviously vulnerable web apps that are being attacked. Lots of developers are now building brilliantly secure software, and their web apps are still getting hit. The sad fact is that if an attacker is good enough, they’ll get through.

it’s not just the obviously vulnerable web apps that are being attacked.

This has us thinking. A lot of the work we do in securing development environments is about helping clients to move beyond a perimeter-focused model of security, and we are now at a stage where our clients usually build fantastic preventative controls into their web apps. Unfortunately, even this work does not fully realize one of the key goals of de-perimeterized security: it doesn’t lead to software that is resilient in the event of an attack.

To achieve resilience, we need web apps that work for incident response teams. We need web apps that allow responders to act with speed and decisiveness. In short, we need applications that have been built with detective controls in mind.

Detective controls

Most well-written web applications already contain input validation, often focused on ensuring the resilience of the application from an operational perspective. For example, many developers will use try-catch function to protect their application from inputs that could cause it to crash. As a result, these error logs tend to be generic, which isn’t a problem most of the time, but it could seriously slow things down for a responder trying to track down an initial access point in the middle of a live incident.

So what can developers do? The simplest answer is for them to provide detective controls in the form of more granular error logs. There are certain signs of attacker reconnaissance and initial access that developers could include in error logs whenever they come up. We’re thinking of SQL inputs with a 1=1 or an unusual character, or syntax for a specific coding language that you wouldn’t expect in a given text field. All of this is basic stuff, but recording it could be invaluable for incident response.

And then there are ways to move beyond the error message. If you ask any incident responder what information they would find useful in a log, they will tell you that recording everything and anything to do with external inputs is going to be useful. Incident response is a bit like police forensic work. The equivalent of a stray hair caught in the fibers of a carpet might be all an incident response team need to catch the attacker red-handed. Any information on who is accessing what and from where is going to be invaluable. Logging the identity of anybody interacting with an application therefore becomes the next logical step.

The snag

Unfortunately, things aren’t quite as simple as we have so far implied.

There are some aspects of user inputs which developers really shouldn’t be trying to record. Web applications developed by enterprise organizations frequently process highly confidential customer data, such as personal identifiers and account numbers. Including these in a log might help incident response, but it’s also going to increase the attack surface by exposing confidential information to the outside. Likewise, logging cookies might provide good traceability, but it potentially gives attackers an easy and privileged route to lateral movement within an estate. And that’s before you factor in the difficulty of using such logs in countries with strict data protection laws such as GDPR or POPIA.

There are some aspects of user inputs which developers really shouldn’t be trying to record.

Then there’s another problem, which is balancing the granularity of your logs with how long you want information to be retained. Chances are, you don’t have unlimited storage, and you are rotating your logs. If you really are recording absolutely everything the application does, this could mean that your logs only reach back a few hours or days. For incident responders, this is almost worse than a sparse and undetailed log. After all, forensic teams are likely to be combing through these logs for evidence of an initial access that occurred days, weeks, or even months before anybody thought to call them in.

The balance

If web app developers want to build detective controls into their logs, they will therefore need to strike a careful balance between including what is useful, and including too much. The simplest way to make a decision about where this balance lies is, we think, by taking yourself through the following set of questions:

This is only a general guide, and you will also have to ask some questions that are specific to your web application. For example, what other information does it ingest that could be useful? Does it ingest host names? LDAP credentials? All of these could be useful for a forensic responder, but they could also pose specific dangers in the event of an attack.

Could detective controls be the future of web applications?

What we are suggesting won’t be easy, at least not at the start. We know that developers are pushed for time, and that building detective controls is always going to be a low priority. This therefore has to be done in the most efficient manner possible, because it is, after all, an insurance tactic to be invested in only after basic vulnerability management is locked down.

The standard Apache access log, for example, contains many of the basic elements we have discussed in this article: developers could do worse than copy its ideas into their web apps.

As such, we hope that with a small effort from a community of developers, open-source libraries could be maintained to cover the most common types of log. This would simplify the process and enable detective controls to become just a small extra step in the development process. Furthermore, developers probably don’t have to look far for examples of how to write code that is useful to responders. The standard Apache access log, for example, contains many of the basic elements we have discussed in this article: developers could do worse than copy its ideas into their web apps.

Ultimately, our proposal is about getting ahead of the attackers. As long as developers put all their security efforts into prevention, they need only make one mistake for an attacker to gain the advantage. By building the foundations of a defensive posture into code, however, developers can take the fight to their adversaries. They can put the attackers on the back foot. They can help create an environment in which it is attackers who need only make one small mistake to be detected. Developers can, in other words, build resilience into their software.

This article was amended on 27/09/2021.

Accreditations & Certificates

F-Secure Consulting (F-Secure Cyber Security (Pty) Ltd) is a level 4 contributor to B-BBEE with a procurement recognition level of 100%. Learn more and download our B-BBEE certificate. Click here to read the press release.

Follow us
@fsecure_consult F-Secure-Consulting f-secure-foundry fsecurelabs