Alfie Champion, Detection Lead
12 mins read
This is a high-level overview for defenders of the research presented by Alfie Champion and James Coote at BlackHat USA 2021. To see the slides from the original presentation, click here.
We call these unusual channels esoteric command and control (C2), and there is good evidence that esoteric channels are now being explored by advanced persistent threat actors. To maintain defenses, it is therefore important to challenge prevailing assumptions about how segregated traditional segregation actually makes critical assets. Without refreshed thinking, capable threat actors will bypass restrictions and 'bridge the gap'.
For security managers, the primary concern is that an attacker’s initial foothold on a standard user endpoint could quickly turn into "objective complete" if critical assets are directly accessible from that endpoint. This article will explore what can be done to prevent such an outcome.
Our research has led us to ask the following questions about the trust boundaries between critical networks.
Ultimately, mitigation usually means ensuring that end users are only provisioned with access to the resources they explicitly require. At a technical level, this can translate to the permissions applied within Active Directory (AD) and internal information repositories such as SharePoint, Confluence, and Stash. At a network level, the hosts a given user can communicate with are the focus.
Broadly speaking, attackers use 2 types of C2: the first for controlling a compromised asset from the internet and the second for communicating internally between compromised hosts.
Fig. 1. Diagram showing the flow of communication between internet-facing C2 and P2P C2 channels
Peer-to-peer (P2P) C2 channels give attackers control of assets without the need to establish a new outbound channel, and therefore allow them to reach areas of the network that don’t have direct internet connectivity, such as a server estate.
For a bi-directional C2 to be effective over an arbitrary medium, certain key actions must be possible. First, the attacker needs to be able to read from the medium. Second, they need to be able to write to it. Finally, depending on the nature of the C2 channel, they may require the ability to list and delete messages.
While the examples in this article come from our own offensive research as part of F-Secure’s red team, esoteric C2 is no longer confined to theory. It has now emerged as a real-world threat:
Rationally, it is therefore likely to be only a matter of time before sophisticated threat actors are also detected using P2P esoteric C2.
In this section, we look at 4 esoteric channels that we successfully used to turn access to segregated networks into a persistent foothold, together with the mitigations that we recommend defenders employ.
VMware's ESXi and vCenter products are near ubiquitous virtualization technologies in corporate environments. Importantly, they offer administrators the ability to manage assets across multiple networks from the same centralized location. These networks could be in development, testing or production zones, or perhaps in OT environments, effectively crossing different levels of an established Purdue model. For an attacker, administrative access to vSphere portals therefore presents a myriad of exploitation opportunities.
Whilst vCenter provides a means to view and interact with the desktop of managed VMs, it does not give attackers a direct means to establish a C2 channel. Network design also frequently prevents communication between the initially compromised host and the target, or the host and the internet.
Fig. 2. Exploitation scenario whereby no connectivity is permitted between source and target hosts, but access permitted to vCenter portal.
This can be seen in the illustration shown, where the attacker has web access to the vSphere portal (denoted by the green HTTPS arrow), but no access to or from a target host.
This is where VMWare Tools comes in: “a suite of utilities that enhances the performance of the virtual machine’s guest operating system and improves management of the virtual machine”.
With the relevant permissions, the attacker can exploit this to upload and execute malware and subsequently share files with a host managed by vCenter. This gives the attacker the ability to read and write to the target host, and with further API calls they can list and delete files to clean up once a message has been delivered. The high-level data flow is summarised below:
Fig. 3. Dataflow between hosts using vCenter API for command and control.
Access to vSphere portals and APIs should be explicitly prevented from standard endpoints and only permitted from dedicated jump hosts or Privileged Access Workstations (PAWs) provisioned for administrative purposes. This is especially important given the recent targeting of virtualized assets by ransomware actors. Remotely accessible VMWare services have also been actively targeted by threat actors, and so under no circumstances should they be exposed to the internet.
As VMWare infrastructure can be used to host Tier 0 assets (e.g., domain controllers), this infrastructure should itself be considered Tier 0. Similarly, the accounts used to administrate virtualized assets should be given the same protections, ensuring dedicated accounts are used, separate from those used for general, day-to-day, non-administrative activities.
Additional hardening may also be applied at a guest host level. VMWare Tools permits the selective installation of its components, allowing essential drivers to be installed on virtual machines, without the Guest Operations features needed to perform C2 as described above.
Enterprise printing is typically handled by dedicated print servers that manage queues of print 'jobs'. Interestingly, a print job can have a name approximately 1MB in size and the job itself does not have to be valid (i.e. printable) for it to be added to the queue.
Printing infrastructure is commonly considered a shared service. It is therefore entirely possible that 2 hosts which are otherwise unable to communicate with each other have access to the same print server. Considering our requirements for a C2 channel, this could allow an attacker to:
There is one limitation to this scenario: both sides of the 'conversation' must be running in the context of the same account, or as an administrator with the ability to otherwise modify the queue. However, this doesn’t provide too much restriction for an attacker, as they may use the same compromised credentials on both hosts. This scenario can be seen in the illustration provided.
Fig. 4. Exploitation scenario whereby source and target hosts can communicate with shared printing infrastructure, but not with each other directly.
Because many end users make use of printers for everyday activities, entirely restricting the ability to submit print jobs is not viable. Instead, provisioning separate printing services for discrete zones would prevent them from being bridged.
There are also opportunities for detection. Print logs can be configured to log the full print job name via group policy at the below location:
It is likely that an attacker using print jobs to transfer arbitrary data would generate longer job names, with greater entropy, than those used for legitimate printing activities.
Fig. 5. Exploitation scenario in which only RDP traffic is permitted to a target host.
Jump hosts are commonly used by administrators to access sensitive assets—such as a production server environment—from the corporate network. In a well segregated environment, connectivity back to the initial host and to the internet would be blocked, even with the necessary credentials and a means to execute arbitrary code.
Attackers can nevertheless establish an esoteric C2 channel through a drive mapped to the target system via Remote Desktop. In this way, they can write messages as files to the mutually accessible drive, such that both the original and target hosts can communicate. The below diagram highlights the dataflow between the two hosts:
Fig. 6. Dataflow between two hosts using a mapped drive as the C2 medium.
The best way to mitigate this C2 channel is to harden the configuration of the target jump host. The mapping of local drives can be prevented entirely via group policy being applied to the target host. Specifically, the below entry should be enabled:
Computer Configuration > Administrative Templates > Windows Components > Remote Desktop Services > Remote Desktop Session Host > Do Not Allow Drive Redirection
This hardening should be applied in conjunction with broader software restriction, provided by solutions such as Windows Defender Application Control. The restricted execution of scripts and executables—among a broader suite of features—can radically reduce an attacker's ability to install malware and establish the C2 channel in the first instance.
Most enterprise organizations make use of Microsoft's Active Directory, either on-premises, in the cloud, or as a hybrid of the two. Crucially, AD is often also used in critical production or OT environments.
Attackers can achieve C2 through Active Directory because of two factors. Firstly, end users can modify several of their Active Directory account's data fields (or 'attributes') via the LDAP protocol. As an example, they can update their phone number. Secondly, any other AD user can then read these attributes, allowing them to read that new phone number. This is, of course, by design.
Notably, only administrators or the user in question can change their attributes. This restricts which accounts can communicate via LDAP C2, though the same account may be reused across multiple hosts in a network.
The key attack surface for this C2 channel arises where the same Active Directory infrastructure is used across both corporate and critical environments. If attackers can communicate with the same domain controller (DC) in both network zones, they can establish a low-latency C2 channel between the two.
Fig. 7. Exploitation scenario whereby source and target hosts can communicate with a single domain controller via LDAP, but with no direct communication with each other.
This C2 channel relies on abuse of legitimate functionality. While some user attributes permit greater data throughput than others and detections for changes to these specific attributes may be explored, the most impactful preventative mitigation is the separation of AD infrastructure between networks.
Notably, separate AD infrastructure for corporate and critical networks also prevents the immediate compromise of sensitive network areas should the corporate network be compromised.
In all but the most sensitive and segregated operating environments, connectivity between network zones exists to facilitate day-to-day deployment, administration, and maintenance. Such connectivity is often a considered business risk that attempts to balance security and usability, but the consequent security risks have perhaps been underestimated.
Our research challenges assumptions of just how connected corporate and production environments may be, and how shared services and infrastructure can be the medium by which a motivated attacker may maintain a persistent foothold within these most sensitive network areas. With Critical National Infrastructure (CNI) now firmly within the purview of ransomware actors, bridging enterprise and OT networks through esoteric C2 will be of interest to many attackers.
There are detection opportunities for all the C2 scenarios we have explored in this article, available in this presentation from BlackHat USA 2021. However, alongside the channel-specific mitigations outlined, the most important action for defenders is to re-evaluate the interconnectivity that exists in their own environments. The key point to address is that unintended connections can exist in any technology and there is no "checklist" of technologies that should be reviewed. As such, security teams working on hardening their network can use this article to: