Do you have all the logs you need?

Josh Gideon, Security Consultant
March, 2018
10 mins read

Security Operation Centers rely solely on log data from various security controls for indicators of abnormal activity within the network. Without the correct logs, a SOC’s capabilities are limited. Does your SOC’s SIEM ingest the right logs?

A key function of a Security Operation Centre (SOC) is aggregation and searching of logs from across the estate, typically into a Security Incident and Event Management (SIEM) tool or other data platform. The ability to detect malicious activity, or to query suspicious activity, is dependent on whether the right data is being collected in the first place. F-Secure Consulting has worked with a number of organizations to develop their SOC’s capability and in our experience many organizations are collecting some, but not all, of the key data sources.

This article covers the key feeds that SOCs should be ingesting into data platforms. These recommendations are based on our experience of the minimum feeds needed to robustly investigate incidents or be alerted to an ongoing incident.

Domain Controller Event Logs

Domain Controllers, the backbone to any Windows domain, house a vast amount of valuable information for defensive teams. Domain Controllers are primarily responsible for authentication across a Windows domain. As such, Domain Controllers log information such as:

  • Active Directory authentication logs;
  • Modifications to AD groups (requires group policy changes).

Domain Controller logs can be utilized for a number of high-fidelity, high-confidence detection use cases, targeted at specific attacker techniques. For example, monitoring for high numbers of failed authentication logs across multiple accounts could be indicative of a brute force attack. Additionally, monitoring high-privileged group membership could provide indications of attackers attempting to privilege escalate or insert high-privileged backdoor accounts.

Endpoint Logs

While Endpoint Detection and Response (explained below) solutions can provide extensive coverage of endpoints, these solutions often have varying visibility limitations that can be complimented with the collection of endpoint logs. A frequent example of this is persistence techniques – some EDR solutions will miss common techniques while a targeted detection can be achieved through monitoring of specific event logs.

Endpoints can generate huge numbers of logs and alerts, so choosing the logs to export to centralized logging should be based on understanding the types of attack and the log artefacts that are produced. The Japan CERT (Computer Emergency and Response Team) has published extensively on log artefacts generated by particular attacks.[1]

Furthermore, endpoint logs can be enriched with additional utilities such as the Microsoft tool Sysmon (Windows only). Sysmon is a lightweight tool that can be configured to target specific system activity, or in this case, specific attacker activity. Sysmon can be configured with a relatively high-confidence alerting ratio, allowing SOCs to develop visualisation dashboards focused on the alerts generated by Sysmon.

Next Step: Endpoint, Detection and Response

Endpoint Detection and Response (EDR) solutions are aimed at monitoring and detecting (occasionally preventing) suspicious or abnormal activity on endpoints, over and above that provided by traditional signature-based technologies such as anti-virus and endpoint logging. An EDR solution, in our opinion, is the most important technology for detection and response of attacker activity on the endpoints.

An effective EDR solution provides comprehensive endpoint visibility over and above that which is provided by traditional Windows event logging.

EDR solutions can range in functionality and effectiveness across different tools. However, common pitfalls in solutions provide pathways for attackers to evade detection and remain hidden within an environment. An EDR solution must, at a minimum, include:

  • Centralized log collection or analysis to provide threat hunting and incident response capabilities;
  • The ability to create detection rules (use cases) either within the EDR appliance or from the logs collected within the SIEM;
  • The ability to query common sources for artefacts of compromise such as process history, registry edits, and network traffic;
  • The ability to read and subsequently detect “in-memory” suspicious activity, such as the execution of Mimikatz. Sophisticated attackers will avoid storing files on disk so as not to raise suspicion and to increase the complexities of detection.

We often witnesses weaknesses within one of the above criteria, more frequently the “in-memory” inspection. However, as attackers employ more sophisticated “in-memory” techniques and actively avoid storing files on disk, the requirement for an EDR to inspect memory is more present that ever.

DNS Logging

Domain Name Service (DNS) is the protocol that translates domain names (URLs) into IP addresses or vice versa. Typical corporate environments will house internal DNS servers for the resolution of internal and external domain names to IP addresses – a valuable logging source comprising of information on different websites accessed by users or endpoints.

Defensive teams can utilise DNS logs for:

  • Monitoring internet access;
  • Identifying if domains that have been reported to be malicious have been accessed from assets in the organization;
  • Monitoring for indications of DNS tunnelling. DNS tunnelling is a technique that allows an attacker to abuse the DNS protocol for external command and control communication. Commonly used in isolated environments where HTTP/S traffic is denied however DNS requests are permitted.

DHCP Logging

Dynamic Host Configuration Protocol (DHCP) is the protocol that automatically assigns IP addresses to devices from a pre-configured address space. In simple terms, when a device connects to a network, the device will perform a DHCP request to the listening DHCP server. The DHCP server will then respond and assign the device with an available IP address, providing the device with the ability to communicate on the network. Most notably, present within the DHCP logs is the device’s MAC address, associated IP, and hostname, which can be crucial in rapidly identifying a device that has been indicated as being compromised.

Monitoring and alerting to unknown and unrecognized devices is also important for most organizations. Physically breaching a target organization and placing a network device can allow an attacker remote administration capabilities and has been seen in real world attacks.[2] As such, immediate detection of these devices is crucial to minimizing the potential impact.

Attackers will often masquerade as everyday workers, maintenance engineers, cleaners or safety inspectors in order to physically breach an organization.

A well-maintained asset register should contain detailed information including MAC addresses and associated hostnames on all devices within the organization. Correlating this information to DHCP logs could provide first indications when a rogue device is connected into the corporate network.

Web Proxy

Most organizations will mandate that web traffic from endpoints traverse a web proxy before accessing the internet. Web proxies provide organizations with visibility of internet traffic and added functionality, such as the ability to granularly control the types/categories of websites users are permitted to access. Enforcement of website categorization can be an effective control against non-targeted malware attacks, preventing the communication to command and control (C2) servers. Although trivial to bypass, this increases the sophistication requirement of an attacker. Furthermore, websites can be immediately blacklisted across an organization – very effective in targeted attacks.

Coupled with prevention, web proxy logging data can be utilized for detective purposes. For example, following the execution of malicious code, most attackers will attempt to establish a C2 channel for remote communication with the compromised endpoint. Commonly, attackers will abuse standard protocols for C2 communication such as HTTP/S. Simply, attackers will attempt to masquerade within “normal” user traffic, so as not to raise suspicion. However, for detection teams, C2 traffic can often be “noisy” and communicate to sites that deviate from the user’s or endpoint’s norm. As such, the creation of dashboards, focused on internet traffic within the SIEM can highlight when an unusual domain is frequently contacted or has abnormally high spikes in traffic. Alternatively, baselining user browsing habits and monitoring for deviations can be effective approaches to detecting suspicious activity.


NetFlow data provides metadata (but not content) of IP traffic as it traverses the network. Such data can provide valuable insight into network connections to and from different systems within the environment. Raw capture of internal traffic is often not possible within organizations owing to the number of tap points that would be required and the amounts of data. However, NetFlow data can often be obtained more easily and as it contains metadata about connections only (attributes like source, destination, amount of data transferred etc.) it takes up far less storage than full packet captures.

Collecting all logs is usually unnecessary. Available logs should be evaluated to determine whether ingestion will increase abilities to detect specific attacker actions.

Following the initial compromise of a user endpoint, attackers will often engage in internal reconnaissance and lateral movement activities as a means of enumerating the environment, understanding targets of interest before moving within the network, working towards a wider objective. Such activities will often deviate from “normal” user and endpoint behaviour and can be indicative of suspicious activity occurring.

Take BloodHound[3] as an example – a common internal reconnaissance tool for visualizing an entire Active Directory structure with added functionality for obtaining active logon session data and more. BloodHound performs LDAP queries against Domain Controllers that are often indistinguishable from normal traffic. However, collection and analysis of NetFlow data can highlight when abnormally large queries occur, which could be indicative of enumeration activities such as BloodHound occurring.

Anti-Malware Email Appliance

Figures retrieved from the 2017 Verizon Data Breach Report[4] suggest that 66% of initial malware infections stemmed from malicious email attachments. In recent years, threat actors ranging from low-sophistication, “script-kiddie” attackers to APT level attacks, have all utilized phishing as the initial delivery mechanism of malicious code into the target organisation. As such, the importance of an email scanning appliance with anti-malware and sandbox capabilities is increasing and it is equally important for SOCs to have visibility of alerts that have been generated.


Anti-virus (AV) is often downplayed within the Information Security industry due to the simplicity of evading it. Attackers of varying levels of sophistication will commonly create new or modify existing malware with the objective of altering signatures and therefore evading detection. However, AV serves a purpose and can be an effective preventative mechanism against common, known malware samples. Moreover, within an organization, deployment of AV signatures can be an immediate mitigation to minimizing the number of malware infections of new, unseen samples.

From an SOC’s perspective, the collection and processing of the majority of AV alerts is unnecessary and can dramatically increase analyst workload with little returns. A large percentage of AV alerts are indications that a known malware sample has been found and subsequently contained or removed from the system – most commonly user endpoints.

However, this does not apply to all AV alerts and, in some cases, can indicate that an attack has reached dangerous stages and requires immediate investigation, such as:

  • AV alerts on assets that are considered critical such as Domain Controllers or Databases holding PPI data;
  • Specifically monitoring critical assets requires maintaining an asset register coupled with a measurement of the criticality of the asset. Implementation of correlation rules within the SIEM can raise the severity of an alert based on the asset;
  • AV alerts that could not be removed from a system for an arbitrary reason;
  • Frequent AV alerts on an individual host;
  • AV alerts considered in the context of other information such as suspicious network connections from a particular host;
  • AV alerts on high-risk malware such as hacking tools.


A large number of log sources will be available to every organization and while some will be consumed, many organizations will have more sources that could be worth considering. Organizations should evaluate the available log sources and determine whether ingestion of such logs could provide further capabilities to detect specific attacker actions.

Organizations should validate that the logs currently ingested sufficiently map to the attack techniques that are expected to be performed against the environment. Purple team assessments can provide a comprehensive overview of an organization’s exposure to specific attack techniques across the cyber kill chain.

Additionally, organizations should continuously validate that log sources are available and that valuable log sources do not cease to operate. SOCs rely solely on log data from various security controls for indicators of abnormal activity within the network - without the logs, a SOC’s capabilities are compromised.

While many security controls that already exist on networks can be used to enrich and supplement the abilities to detect attackers, other tooling (both free and paid for) exist that can supplement the existing logs.





Accreditations & Certificates

F-Secure Consulting (F-Secure Cyber Security (Pty) Ltd) is a level 4 contributor to B-BBEE with a procurement recognition level of 100%. Learn more and download our B-BBEE certificate. Click here to read the press release.

Follow us
@fsecure_consult F-Secure-Consulting f-secure-foundry fsecurelabs