Investigating a GKE Container

By

As containerized applications have gained popularity and are the most common method for deploying applications in the cloud, digital forensics analysts need to be familiar with the process and tools to examine the containers. 

A previous blog post Container Forensics with Docker Explorer dives into details on analyzing a compromised container with an open source tool Docker Explorer. 


This article focuses on analysis of Google Kubernetes Engine (GKE) containers running containerd and the process and open source tools that would aid in forensic analysis of a potential compromised container. The research work for this article was conducted using a GKE cluster. In this article, the commands starting with gcloud are Google Cloud Platform (GCP) specific commands, and kubectl commands are Kubenetes commands that are the same for all cloud providers.


In our investigation, we received signals indicating scanning activity towards the web server running on a GKE cluster. All we know at this stage is the target was a GKE cluster called wp-cluster in an analysis-lab-340016 project. 


In order to start our analysis, let us assume the admin team provisioned the rights to the wp-cluster, and the investigation team has their own cleverly named project called response-project.


You may want  to start cloning everything in the cluster to unravel the mystery, but this may not scale very well for two reasons:

  1. A Kubernetes cluster can have a large number of nodes.

  2. Forensic tools used may not be container aware.


Instead, let’s first find the right instance.

Kubernetes Cluster

A Kubernetes cluster may contain several nodes; Nodes are the virtual machines that run the pods and containers. Pods are the smallest deployable objects in a Kubernetes, and containers are where containerized applications run.


For more information about Kubernetes cluster architecture, see Cluster Architecture | Kubernetes.


Let’s start exploring the GKE cluster wp-cluster using kubectl; kubectl is the command line utility that interacts with the Kubernetes API server and provides runtime information to the analyst.


Before using kubectl, we must authenticate to GCP and gain permissions for the GKE cluster wp-cluster. It is best to have the following information in advance:

  • Cluster name i.e. wp-cluster

  • Project name i.e. analysis-lab-340016

  • Zone i.e. australia-southeast2-b


The figure below shows the commands used to authenticate to GCP, and get access to wp-cluster. If your cluster is on a different cloud provider, you can follow the cloud provider specific authentication commands.


$ gcloud auth login

$ gcloud auth application-default login

$ gcloud container clusters get-credentials wp-cluster --zone australia-southeast2-b --project analysis-lab-340016


Let’s start with basic information about the wp-cluster. The figure below shows the basic cluster information.


$ kubectl cluster-info

Kubernetes control plane is running at https://34.129.142.118

GLBCDefaultBackend is running at https://34.129.142.118/api/v1/namespaces/kube-system/services/default-http-backend:http/proxy

KubeDNS is running at https://34.129.142.118/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

Metrics-server is running at https://34.129.142.118/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

 

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.


Kubernetes Namespaces

In Kubernetes, namespaces provide a mechanism for isolating groups of resources within a single cluster.


A standard cluster in GKE has the following namespaces:

  • default - The default namespace for objects with no other namespace

  • kube-system - The namespace for objects created by the Kubernetes system

  • kube-public - The namespace reserved for cluster usage, and is readable by everyone without authentication

  • kube-node-lease - The namespace for the lease objects associated with each node


For more information about Kubernetes namespaces, see Namespaces | Kubernetes.

Kubernetes Nodes

Kubernetes nodes are virtual machines in a cluster. A standard configuration in GKE generates three nodes for a cluster. 


Let’s use the kubectl command to list the nodes in wp-cluster.


$ kubectl get nodes

NAME                                        STATUS   ROLES    AGE     VERSION

gke-wp-cluster-default-pool-b4e5d97b-2fn8   Ready    <none>   7d12h   v1.21.6-gke.1500

gke-wp-cluster-default-pool-b4e5d97b-7fkt   Ready    <none>   7d12h   v1.21.6-gke.1500

gke-wp-cluster-default-pool-b4e5d97b-btxm   Ready    <none>   7d12h   v1.21.6-gke.1500


Kubernetes Pods

Kubernetes pods are the smallest deployable objects in Kubernetes that run containers. A pod generally has a single container; However, some pods may contain more than one container.


For more information about pods, see Pod | Kubernetes Engine Documentation | Google Cloud.


Let’s use the kubectl command to list the pods.


$ kubectl get pods

NAME                         READY   STATUS      RESTARTS   AGE

wordpress-1-deployer-d4w9s   0/1     Completed   0          7d12h

wordpress-1-mysql-0          2/2     Running     0          7d12h

wordpress-1-wordpress-0      2/2     Running     2          7d12h


The command above does not show information about the node on which these pods are running. Use the kubectl get pods -o wide command to show pod and associated node information.


If your investigation requires inspecting resource utilizations, which is typical for coin miner investigations, use the kubectl top command to view resource utilization.


$ kubectl top node

NAME                                        CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%

gke-wp-cluster-default-pool-b4e5d97b-2fn8   115m         12%    1326Mi          47%

gke-wp-cluster-default-pool-b4e5d97b-7fkt   61m          6%     630Mi           22%

gke-wp-cluster-default-pool-b4e5d97b-btxm   134m         14%    1663Mi          59%


In a general coin miner compromised node, the CPU utilization is very high. In the figure above, the CPU utilization looks normal which is a preliminary indication of no coin miner on the nodes. However, under certain circumstances, coin miner may be in a dormant state, thus comprehensive analysis is necessary.

Kubernetes Containers

Kubernetes containers are where containerized applications run and are immutable by design. For more information about Kubernetes containers, see Containers | Kubernetes.


In this case, let's list all the containers running in the cluster. Unlike listing nodes and pods, there is no kubectl get containers command to show containers. But, a couple of kubectl commands can provide some insight about the containers.


  1. Using kubectl top pod command


$ kubectl top pod --containers

POD                       NAME              CPU(cores)   MEMORY(bytes)

wordpress-1-mysql-0       mysql             8m           228Mi

wordpress-1-mysql-0       mysqld-exporter   5m           5Mi

wordpress-1-wordpress-0   apache-exporter   1m           8Mi

wordpress-1-wordpress-0   wordpress         8m           90Mi


In the figure above, the column NAME shows the container name.


  1. Using kubectl get pods command


$ kubectl get pods -o 'custom-columns=NAMESPACE:.metadata.namespace,POD:.metadata.name,CONTAINERS:.spec.containers[*].name'

NAMESPACE   POD                          CONTAINERS

default     wordpress-1-deployer-d4w9s   deployer

default     wordpress-1-mysql-0          mysql,mysqld-exporter

default     wordpress-1-wordpress-0      wordpress,apache-exporter


In the figure above, the column CONTAINERS shows comma separated container names in a pod.


NOTE: The containers shown above are only for Kubernetes default namespace default. Use --all-namespaces flag to show containers for all namespaces in the cluster.

Containers of Interest

The output of kubectl top pod --containers command showed that the cluster has three pods and four containers. Of the four containers, let us find out the containers that are exposed to the internet. In Kubernetes, containers are exposed to the internet using Kubernetes services. For more information about Kubernetes services, see Service | Kubernetes. 


The kubectl get services command shows the services in the cluster. The figure shows the output of the command.


$ kubectl get services

NAME                              TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)        AGE

kubernetes                        ClusterIP      10.64.16.1     <none>          443/TCP        7d19h

wordpress-1-apache-exporter-svc   ClusterIP      None           <none>          9117/TCP       7d19h

wordpress-1-mysql-svc             ClusterIP      None           <none>          3306/TCP       7d19h

wordpress-1-mysqld-exporter-svc   ClusterIP      None           <none>          9104/TCP       7d19h

wordpress-1-wordpress-svc         LoadBalancer   10.64.29.229   34.129.29.205   80:32221/TCP   7d19h


In the output above, we should look for TYPE LoadBalancer, and EXTERNAL-IP containing a public IP address. The service wordpress-1-wordpress-svc matches the pattern and has the public IP address 34.129.29.205, which provides access to the wordpress site on the internet.


Let’s get the detailed information about the service wordpress-1-wordpress-svc using the kubectl describe service command.


$ kubectl describe svc wordpress-1-wordpress-svc

Name:                     wordpress-1-wordpress-svc

Namespace:                default

Labels:                   app.kubernetes.io/component=wordpress-webserver

                          app.kubernetes.io/name=wordpress-1

Annotations:              cloud.google.com/neg: {"ingress":true}

Selector:                 app.kubernetes.io/component=wordpress-webserver,app.kubernetes.io/name=wordpress-1

Type:                     LoadBalancer

IP Family Policy:         SingleStack

IP Families:              IPv4

IP:                       10.64.29.229

IPs:                      10.64.29.229

LoadBalancer Ingress:     34.129.29.205

Port:                     <unset>  80/TCP

TargetPort:               80/TCP

NodePort:                 <unset>  32221/TCP

Endpoints:                10.56.0.6:80

Session Affinity:         None

External Traffic Policy:  Cluster

Events:                   <none>


The output above provides additional information about pod, container, and application:

  • Labels: app.kubernetes.io/name=wordpress-1 - Kubernetes application name

  • IPs: 10.64.29.229 - IP addresses of the containers associated with the service

  • Endpoints: 10.56.0.6:80 - IP address and port of the pods associated with the service


For more about Kubernetes labels, see Recommended Labels | Kubernetes.


Based on the service details above, the service is associated with a single pod and a single container. The output of the kubectl get pods -o wide command provides the IP addresses of the pods; 10.56.0.6 is the IP address of the pod wordpress-1-wordpress-0.


Let's get details about the pod using the kubectl describe pod command. The output of the command is verbose. Thus, the figure below shows only the relevant section of the output.


$ kubectl describe pod wordpress-1-wordpress-0

Name:         wordpress-1-wordpress-0

Namespace:    default

Node:         gke-wp-cluster-default-pool-b4e5d97b-btxm/10.192.0.12

<SNIP>

IP:           10.56.0.6

IPs:

  IP:           10.56.0.6

Controlled By: StatefulSet/wordpress-1-wordpress

Containers:

  wordpress:

    Container ID:   containerd://b0b4c011cce6ba42f515416f965ab7cb0c7c9de7a469a5da5768ce6404298b24

<SNIP>

    Port:           80/TCP

<SNIP>

    Mounts:

      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ngpvw (ro)

      /var/www/html from wordpress-1-wordpress-pvc (rw,path="wp")

      /var/www/html/.htaccess from apache-config (ro,path=".htaccess")

      /wp-install.sh from config-map (ro,path="wp-install.sh")

  apache-exporter:

    Container ID:  containerd://3544209cfda893703458d7d0a6a65970bfb46e9be6a60faa1e4e9d0adae11b55

<SNIP>

    Port:          9117/TCP

<SNIP> 

   Mounts:

      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ngpvw (ro)


So far in the investigation, we have identified the following:

  • Application name - wordpress-1

  • Container IP address - 10.64.29.229

  • Container name - wordpress (based on container port)

  • Pod IP address and name - 10.56.0.6:80, wordpress-1-wordpress-0

  • GKE node name - gke-wp-cluster-default-pool-b4e5d97b-btxm

Container Log

In the previous section, we have narrowed down the pod and the container we are interested in investigating further. Kubernetes provides the kubectl log command that prints the container logs on screen. The command below redirects the logs to a file on disk.


$ kubectl logs wordpress-1-wordpress-0 -c wordpress > wordpress-1-wordpress-0_wordpress.log


The output generated by kubectl log command includes the Apache access log. The figure below shows the partial content of wordpress-1-wordpress-0_wordpress.log containing sqlmap activities.


<SNIP>

 

10.56.0.1 - - [09/Feb/2022:21:17:58 +0000] "GET /wp-login.php?redirect_to=http%3A%2F%2F34.129.29.205%2Fwp-admin%2F HTTP/1.1" 200 2514 "-" "sqlmap/1.6#stable (https://sqlmap.org)"

10.192.0.14 - - [09/Feb/2022:21:18:18 +0000] "GET /wp-login.php?redirect_to=http%3A%2F%2F34.129.29.205%2Fwp-admin%2F&gdgF=4627%20AND%201%3D1%20UNION%20ALL%20SELECT%201%2CNULL%2C%27%3Cscript%3Ealert%28%22XSS%22%29%3C%2Fscript%3E%27%2Ctable_name%20FROM%20information_schema.tables%20WHERE%202%3E1--%2F%2A%2A%2F%3B%20EXEC%20xp_cmdshell%28%27cat%20..%2F..%2F..%2Fetc%2Fpasswd%27%29%23 HTTP/1.1" 200 2514 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.458.1 Safari/534.3"

10.56.0.1 - - [09/Feb/2022:21:18:18 +0000] "GET /wp-login.php?redirect_to=http%3A%2F%2F34.129.29.205%2Fwp-admin%2F HTTP/1.1" 200 2514 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.458.1 Safari/534.3"

10.192.0.14 - - [09/Feb/2022:21:18:19 +0000] "GET /wp-login.php?redirect_to=5714 HTTP/1.1" 200 2514 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.458.1 Safari/534.3"

10.192.0.14 - - [09/Feb/2022:21:18:19 +0000] "GET /wp-login.php?redirect_to=http%3A%2F%2F34.129.29.205%2Fwp-admin%2F.%28%22%27%28%29%28%29%28%28 HTTP/1.1" 200 2533 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US) AppleWebKit/534.3 (KHTML, like Gecko) Chrome/6.0.458.1 Safari/534.3"

 

<SNIP>


The Apache access log above omits client source IP address. The default configuration does not show the client's source IP address; the behavior can be changed by updating Kuberenetes configuration. 


For more information about preserving client source IP addresses, see Create an External Load Balancer | Kubernetes.

Analyzing a Node

In the section Containers of Interest, we identified the GKE node that contains the suspected compromised container. In the next section, we will make a copy  of the GKE node and analyze the suspected compromised container.

Disk Acquisition

There are a number of ways to take a copy of the GKE node; the common options are using Google Cloud Console, Google Cloud API, gcloud command etc. However, we will use a forensic tool called dftimewolf. 


For more information about dfTimewolf and source code, see dfTimewolf and GitHub - log2timeline/dftimewolf.


The advantage of using dftimewolf is that it has the recipe gcp_forensics which completes all the required tasks for us. The gcp_forensic recipe:

  1. Copies a GCP image of the GKE node to a separate analysis GCP project

  2. Creates an analysis VM in the analysis GCP project 

  3. Attaches the GCP image as a READ_ONLY disk to the analysis VM


The gcp_forensics recipe takes a required argument (source project name) and optional arguments. You can learn about the arguments required for the recipe by using the recipe specific help command i.e. dftimewolf gcp_forensics -h. The figure below shows the dftimewolf recipe used to collect forensic evidence.


$ dftimewolf gcp_forensics --analysis_project_name response-project --instance gke-wp-cluster-default-pool-b4e5d97b-btxm --all_disks --zone australia-southeast2-b analysis-lab-340016


NOTE: Before running the dftimewolf command, you must run gcloud auth application-default login. 


Let’s breakdown the dftimewolf’s gcp_forensics arguments:

  • --analysis_project_name: Name of the project where the analysis VM will be created and disks copied to. For this investigation, the project name is response-project.

  • --instance: Name of the instance to analyze. For this investigation, the instance name is gke-wp-cluster-default-pool-b4e5d97b-btxm.

  • --all_disks: Copy all disks in the designated instance.

  • --zone: The GCP zone where the analysis VM and copied disks will be created. For this investigation, the zone is australia-southeast2-b.

  • Required positional argument remote project name: Name of the project containing the instance / disks to copy. For this investigation, the remote project is analysis-log-340016.


The console output of dftimewolf recipes are verbose. It is very tempting to glance and skip the console messages. However, the console messages contain few critical information for the next steps in the investigation and they are:

  • Analysis VM name: gcp-forensics-vm-20220211200048

  • Name of the evidence disks:

    • evidence-gke-wp-cluster-default-pool-b4e5d97b-btx-aa530db5-copy

    • evidence-gke-wp-cluster-550c00f-pvc-4c86e516-bad5-6cb054cb-copy

    • evidence-gke-wp-cluster-550c00f-pvc-261b3d80-ba6e-5d24f409-copy

    • evidence-gke-wp-cluster-550c00f-pvc-d483df58-6153-9c1f6111-copy


The figure below shows the partial output of dftimewolf gcp_forensics recipe.


<SNIP>

[2022-02-11 20:00:48,346] [GoogleCloudCollector] SUCCESS  Your analysis VM will be: gcp-forensics-vm-20220211200048

<SNIP>

[2022-02-11 20:03:09,730] [libcloudforensics.providers.gcp.forensics] INFO     Disk gke-wp-cluster-default-pool-b4e5d97b-btxm successfully copied to evidence-gke-wp-cluster-default-pool-b4e5d97b-btx-aa530db5-copy

<SNIP>

[2022-02-11 20:04:42,136] [GoogleCloudCollector] SUCCESS  Disk gke-wp-cluster-550c00f-pvc-4c86e516-bad5-4cb3-932f-c755daa49d92 successfully copied to evidence-gke-wp-cluster-550c00f-pvc-4c86e516-bad5-6cb054cb-copy

<SNIP>

[2022-02-11 20:05:55,391] [GoogleCloudCollector] SUCCESS  Disk gke-wp-cluster-550c00f-pvc-261b3d80-ba6e-447a-8d36-44dd2a1368fd successfully copied to evidence-gke-wp-cluster-550c00f-pvc-261b3d80-ba6e-5d24f409-copy

<SNIP>

[2022-02-11 20:07:09,805] [GoogleCloudCollector] SUCCESS  Disk gke-wp-cluster-550c00f-pvc-d483df58-6153-4c81-86fa-bccb4d3c0a1b successfully copied to evidence-gke-wp-cluster-550c00f-pvc-d483df58-6153-9c1f6111-copy

File system  analysis

The dftimewolf output shows the analysis VM name is gcp-forensics-vm-20220211200048 and the forensic images are attached as READ_ONLY disks to the analysis VM.


  1. Let’s first verify that we can see the analysis VM in the response team’s project (response-project).


$ gcloud compute --project response-project instances list

NAME                             ZONE                    MACHINE_TYPE   PREEMPTIBLE  INTERNAL_IP  EXTERNAL_IP     STATUS

gcp-forensics-vm-20220211200048  australia-southeast2-b  e2-standard-4               10.192.0.2   34.129.167.191  RUNNING


  1. Let’s verify the evidence is attached to the analysis VM.


$ gcloud compute --project=response-project instances list --filter='gcp-forensics-vm-20220211200048' --format="yaml(name,zone,disks[].deviceName)"

---

disks:

- deviceName: persistent-disk-0

- deviceName: evidence-gke-wp-cluster-default-pool-b4e5d97b-btx-aa530db5-copy

- deviceName: evidence-gke-wp-cluster-550c00f-pvc-4c86e516-bad5-6cb054cb-copy

- deviceName: evidence-gke-wp-cluster-550c00f-pvc-261b3d80-ba6e-5d24f409-copy

- deviceName: evidence-gke-wp-cluster-550c00f-pvc-d483df58-6153-9c1f6111-copy

name: gcp-forensics-vm-20220211200048

zone: https://www.googleapis.com/compute/v1/projects/response-project/zones/australia-southeast2-b


  1. Let’s logon to the analysis VM.


$ gcloud compute --project response-project ssh gcp-forensics-vm-20220211200048


  1. Let’s check that the evidence is attached to the analysis VM.


$ ls /dev/sd*

/dev/sda  /dev/sda1  /dev/sda14  /dev/sda15  /dev/sdb  /dev/sdb1  /dev/sdb10  /dev/sdb11  /dev/sdb12  /dev/sdb2  /dev/sdb3  /dev/sdb4  /dev/sdb5  /dev/sdb6  /dev/sdb7  /dev/sdb8  /dev/sdb9  /dev/sdc  /dev/sdd  /dev/sde


  1. Let’s get the details about the partitions. 

Based on the number of partitions for /dev/sdb, it is the boot image of gke-wp-cluster-default-pool-b4e5d97b-btxm. 


$ sudo fdisk -l /dev/sdb

Disk /dev/sdb: 10 GiB, 10737418240 bytes, 20971520 sectors

Units: sectors of 1 * 512 = 512 bytes

Sector size (logical/physical): 512 bytes / 4096 bytes

I/O size (minimum/optimal): 4096 bytes / 4096 bytes

Disklabel type: gpt

Disk identifier: 7C818738-EDF0-B246-960D-0E7EE8655B06

 

Device       Start      End  Sectors  Size Type

/dev/sdb1  8704000 20971486 12267487  5.9G Linux filesystem

/dev/sdb2    20480    53247    32768   16M ChromeOS kernel

/dev/sdb3  4509696  8703999  4194304    2G ChromeOS root fs

/dev/sdb4    53248    86015    32768   16M ChromeOS kernel

/dev/sdb5   315392  4509695  4194304    2G ChromeOS root fs

/dev/sdb6    16448    16448        1  512B ChromeOS kernel

/dev/sdb7    16449    16449        1  512B ChromeOS root fs

/dev/sdb8    86016   118783    32768   16M Linux filesystem

/dev/sdb9    16450    16450        1  512B ChromeOS reserved

/dev/sdb10   16451    16451        1  512B ChromeOS reserved

/dev/sdb11      64    16447    16384    8M BIOS boot

/dev/sdb12  249856   315391    65536   32M EFI System


Typically for Container-Optimized OS (COS), the first partition (in the analysis VM /dev/sdb1) is the writable partition which is mounted as /mnt/stateful_partition. This is the partition where the data of the containers resides.


Next we mount /dev/sdb1 to a mount point /mnt/disks/sdb. Although the evidence disks are attached to the analysis VM as READ_ONLY disks, it is best practice to mount the disks in READ_ONLY mode in the analysis VM.


$ sudo mount -o ro,noload,noexec /dev/sdb1 /mnt/disks/sdb

$ mount | grep /mnt/disks/sdb

/dev/sdb1 on /mnt/disks/sdb type ext4 (ro,noexec,relatime,norecovery)


You may want to run tools like plaso on the mount point /mnt/disks/sdb, which will find all the files on the node including files in the containers. But, you will lose the container context on those files i.e. which containers used those files. In this investigation, we are particularly interested in analyzing the container called wordpress.


Containers in the latest version of GKE use containerd as the container runtime, and containerd data is located at /var/lib/containerd. We will use a tool called Container Explorer (https://github.com/google/container-explorer) to mount the wordpress container for analysis.


Let’s download a pre-compiled container-explorer binary to the analysis VM.


$ wget https://github.com/google/container-explorer/files/8027588/container-explorer-0.0.2-20220209.gz

$ gunzip container-explorer-0.0.2-20220209.gz

$ chmod +x container-explorer-0.0.2-20220209


Let’s also download a yaml file called supportcontainer.yaml; this file contains information that is used by container-explorer to suppress the details of the containers created by Kubernetes system. Using supportcontainer.yaml, we can focus on the containers that are relevant for investigation.


$ wget https://raw.githubusercontent.com/google/container-explorer/main/supportcontainer.yaml


Now let’s use container-explorer to find information about containers. The figure below shows the partial output of the command.


$ sudo ./container-explorer-0.0.2-20220209 --support-container-data supportcontainer.yaml -i /mnt/disks/sdb/ list containers


Let’s breakdown the arguments used in container-explorer:

  • --support-container-data: This file contains information about the containers created by the Kubernetes system.

  • -i: Mount point where the container host system’s image is mounted.

  • list containers: container-explorer command to list the containers on the specified image mount point.


The list containers command provides the following information:

  • NAMESPACE: containerd namespace. For GKE the namespace is k8s.io.

  • TYPE: Docker container or a containerd container. This is useful when a host has both Docker and containerd containers.

  • CONTAINER ID: A unique container identifier

  • CONTAINER HOSTNAME: This field attempts to provide container hostname. This field may show the pod name.

  • IMAGE: Container image

  • CREATED AT: The timestamp when this container was created.

  • PID: Containerized application process ID.

  • STATUS: Container status RUNNING, PAUSED, STOPPED, or UNKNOWN.

  • LABELS: Container labels.


The figure below shows the output of container-explore list containers command.


NAMESPACE       TYPE            CONTAINER ID                                                            CONTAINER HOSTNAME      IMAGE                                                                                                                                   CREATED AT            PID     STATUS  LABELS

k8s.io          containerd      3544209cfda893703458d7d0a6a65970bfb46e9be6a60faa1e4e9d0adae11b55        wordpress-1-wordpress-0 gcr.io/cloud-marketplace/google/wordpress/apache-exporter:5.8.1-20211128-160022                                                         2022-02-04T21:55:28Z  0       UNKNOWN io.cri-containerd.kind=container,io.kubernetes.container.name=apache-exporter,io.kubernetes.pod.name=wordpress-1-wordpress-0,io.kubernetes.pod.namespace=default,io.kubernetes.pod.uid=3f396f5d-77ba-43e8-806c-5c23bfd95ec6


Looking at the output of the container-explorer list containers command, the output did not contain the container name wordpress. This is because container-explorer extracts information directly from the containerd database which uses container ID instead of container name. 


If you recall the output of the kubectl describe pod wordpress-1-wordpress-0 command, it provided the container ID of the wordpress container. The figure below shows the partial output of the command.


<SNIP>

 

Containers:

 wordpress:

   Container ID:   containerd://b0b4c011cce6ba42f515416f965ab7cb0c7c9de7a469a5da5768ce6404298b24

   Image:          gcr.io/cloud-marketplace/google/wordpress:5.8.1-20211128-160022

   Image ID:       gcr.io/cloud-marketplace/google/wordpress@sha256:f9b1b9f82d45f67ef8bc2899cd363a1b42c037660749507f9c9c1100d98fc42b

   Port:           80/TCP

   Host Port:      0/TCP

   State:          Running

 

<SNIP>


With the knowledge of container ID (b0b4c011cce6ba42f515416f965ab7cb0c7c9de7a469a5da5768ce6404298b24

), we can get the container's details using the container-explorer info container command. The figure below shows the full command.


$ sudo ./container-explorer-0.0.2-20220209 --support-container-data supportcontainer.yaml -i /mnt/disks/sdb -n k8s.io info container b0b4c011cce6ba42f515416f965ab7cb0c7c9de7a469a5da5768ce6404298b24


NOTE: The switch -n k8s.io in the above command is a containerd namespace. You may have noticed that the namespace k8s.io does not match with the namespace names we observed in the GKE Cluster section. This is because the namespaces observed while using kubectl command are Kubernetes namespaces, and the namespace used by container-explorer is containerd namespace.


The container-explorer info container command shows verbose information about the container and its specification. The following JSON key provides the useful information about the container:

  • .Labels - Provides pod namespace, name, UID, and container name

  • .CreatedAt - Container creation timestamp

  • .Spec - Open Container Initiative (OCI) runtime specification

  • .Spec.env - Environmental variables used to store configuration key-value


We have learned a little more about the container specification. Now, let’s mount the container to path /mnt/container for analysis. We will use the container-explorer mount command for this.


$ sudo ./container-explorer-0.0.2-20220209 --support-container-data supportcontainer.yaml -i /mnt/disks/sdb --namespace k8s.io mount b0b4c011cce6ba42f515416f965ab7cb0c7c9de7a469a5da5768ce6404298b24 /mnt/container/


Let’s verify that the container was successfully mounted to /mnt/container.


$ mount | grep /mnt/container

overlay on /mnt/container type overlay (ro,relatime,lowerdir=/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/187/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/169/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/168/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/167/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/166/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/165/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/164/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/163/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/162/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/161/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/160/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/159/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/158/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/157/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/156/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/155/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/154/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/153/fs:/mnt/disks/sdb/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/140/fs,xino=off)


Now that we have successfully mounted the container to /mnt/container. You can run your favorite forensic software such as plaso on the /mnt/container.

Conclusion

Kubernetes helps in management and scaling of containerized applications. It introduces several new concepts like nodes, pods, containers, and containerized applications and their relationships. In Kubernetes, a containerized application may run in any node available in the cluster; it is not as simple as in traditional server environments. Identifying which container to investigate may be the first challenge we face when investigating a Kubernetes cluster.

 

In the example above, we chose a simple GKE cluster and demonstrated the way to identify a compromised container with the help of kubectl command. We also demonstrated the forensics tools like dftimewolf and container-explorer that could speed up container  examination.

References

Comments

Post a Comment

Popular posts from this blog

Parsing the $MFT NTFS metadata file

By
Background Numerous articles have been written about parsing the $MFT NTFS metadata file before. Some of these explain the technique itself [ 1 ] and others mostly focus on how to use tools. Rarely do articles mention the pros and cons of the technique from a digital forensics perspective [ 2 ].  What is $MFT parsing?  For readers not familiar with the NTFS file system, MFT stands for Master File Table. This table is stored in a file system metadata file, meaning that as far as the file system is concerned it is a file, however that the file contains file system metadata not “regular” content. The name of this file is “$MFT”.  The MFT consists of a sequence of (predetermined) fixed-size entries (or MFT entries). These entries are typically 1024 bytes in size, however the size is defined in the NTFS volume header and in a “regular” MFT entry itself.  The first 4-bytes of a “regular” MFT-entry (or record) starts with the signature “FILE”. The MFT entry can also be filled wit
Read more

Incident Response in the Cloud

By
Many organizations have begun moving a majority of their services towards the cloud in recent years. As a result, attackers have shifted their focus towards the cloud. This has resulted in new techniques and methods specifically designed to compromise cloud infrastructure, like the recent SALTSTACK vulnerability that was widely exploited [ 1 ]. Therefore it is critical for these organizations to have an incident response team that understands the new risks attached to cloud and how cloud can make incident response easier or harder.  In this blog post we will walk you through each phase you may encounter in traditional incident response and highlight the differences when adopting cloud computing. It is aimed towards both those who are new to incident response and cloud computing. We’ve included insights that will benefit organizations taking that step towards cloud who want to ensure they are prepared to respond efficiently to cloud incidents.  Traditional Incident Response Before jump
Read more

Container Forensics with Docker Explorer

By Anonymous
  Introduction As previous blog posts on cloud forensics have noted, applications are increasingly being deployed using container orchestration frameworks such as Kubernetes, especially in cloud environments.  Similar to traditional deployments on physical servers or virtual machines (VMs), when a containerised application has a security issue it can lead to a compromise of the underlying compute architecture. In the case of container deployments this means a compromise of the container itself, the container host, or even a wider cluster compromise via abuse of orchestration tools. Often digital forensics is required to establish what went wrong and remediate any issues. This article will provide an introduction to container forensics with Docker Explorer by working through a scenario involving a compromised container running within a Kubernetes cluster. Although Kubernetes is briefly mentioned, this article will focus on analysis of an individual container rather than the wider clust
Read more