An update on the latest Turbinia features

By

The Turbinia development team are excited to announce the release of a number of new features and improvements for Turbinia. With the latest updates, Turbinia now includes an API server and a web interface, providing even more flexibility and ease of use. In this blog post we'll take a closer look at these exciting new additions.


Turbinia API Server

Overview

The Turbinia API server is an HTTP REST API that allows clients to submit forensic processing requests and monitor their progress. This opens up new possibilities for integrating Turbinia into other applications or systems, automating forensic processing, and streamlining investigations. 

dfTimewolf–a forensics workflow orchestration tool– is now integrated with Turbinia's API server to create new evidence processing flows.

Features

The API server provides several endpoints to perform different actions. At time of writing this article, the API server supports the following actions:

  • Create custom Turbinia requests (e.g. using recipes or custom POST data parameters)

  • Retrieve the status of requests and their associated tasks to track their progress

  • Retrieve server configuration settings

  • Retrieve a list of enabled Turbinia job types

  • Download request and task output files, such as Plaso storage files

  • Download worker and task log files for easier troubleshooting

We built the API server using OpenAPI as a standard and FastAPI as a framework. It is designed to be flexible and extensible, which allows you to define your own forensic tasks as JSON documents, and more easily integrate Turbinia into existing workflows or systems. Until now, the only ways to integrate Turbinia with other tools was to import a Python client directly in code, or turbiniactl cli tool. The API server and client library make integration easier through HTTP requests, and a lightweight Python package without being required to install all of Turbinia's components and dependencies.

To protect sensitive endpoints and forensic data, the server supports authentication via OAuth2 or OpenID Connect (OIDC) identity providers through oauth2-proxy. This allows for single sign-on through different identity providers like Google, AWS, Okta, etc.

API documentation and a web-based client is available at /docs. The docs UI can be used to test and interact with the different endpoints.

Figure 1. Turbinia documentation and REST client

Interacting with the API

In addition to the built-in web-based client, we created two new packages which are available via PyPi (a Python package management platform accessible via the pip command). A new Python library turbinia-api-lib that can be used to interact with the API server programmatically, and a new command-line tool distributed as turbinia-client which provides a convenient way to interact with the Turbinia API server from the command line. 

The turbinia-client package offers a range of functionality that enables you to submit tasks, monitor their progress and retrieve available results. User documentation and usage examples can be found here.

Usage examples

Available commands can be listed by running the following command:

turbinia-client -h


Options:

  -c, --config_instance TEXT  A Turbinia instance configuration name.

                              [default: (dynamic)]

  -p, --config_path TEXT      Path to the .turbinia_api_config.json file..

                              [default: (dynamic)]

  -h, --help                  Show this message and exit.


Commands:

  config  Get Turbinia configuration.

  jobs    Get a list of enabled Turbinia jobs.

  result  Get Turbinia task or request results.

  status  Get Turbinia request/task status.

  submit  Submit new requests to the Turbinia API server.


To retrieve the status of an existing Turbinia request we would invoke the status command, and the request sub-command as follows:

turbinia-client status request -j 65c873083075409faa6c70f5bb30c6b6

The -j option indicates we want JSON output instead of the default Markdown status report.

{

  "failed_tasks": 0,

  "last_task_update_time": "2023-05-10T21:42:36.788874Z",

  "queued_tasks": 0,

  "reason": "case123",

  "request_id": "65c873083075409faa6c70f5bb30c6b6",

  "requester": "user1",

  "running_tasks": 0,

  "status": "successful",

  "successful_tasks": 34,

"task_count": 34,

  "tasks": [

    {

      "all_args": "",

      "evidence_name": "GoogleCloudDisk:/p15",

      "group_id": "86580b15215241b89ddb3d07ab52bffb",

      "group_name": "",

      "id": "e1d92289f830486c9bce76cd72643fbe",

      "instance": "osdfir-cluster",

      "job_id": "828e6f77c78841329e05ff45ee4ce398",

      "last_update": "2023-05-10T20:32:38.757347Z",

      "name": "ContainerdEnumerationTask",

      "reason": "case123",

      "report_data": null,

      "report_priority": 50,

      "request_id": "65c873083075409faa6c70f5bb30c6b6",

      "requester": "user1",

      "run_time": 0.051558,

      "saved_paths": [

        "/mnt/turbiniavolume/output/65c873083075409faa6c70f5bb30c6b6/1683750749-e1d92289f830486c9bce76cd72643fbe-ContainerdEnumerationTask/worker-log.txt"

      ],

      "status": "Found 0 containers",

      "successful": true,

      "worker_name": "turbinia-worker-7ddc986664-pcrzg"

    },

Turbinia Web UI

We are also excited to release a top requested feature, a new web interface for Turbinia. The web interface provides a user-friendly way of interacting with Turbinia's new features. It offers a visual representation of the processing tasks, their progress, and results in a web-based dashboard. This makes it easy for investigators to monitor forensic processing tasks, view results, and track them in real-time.

Figure 2. Screenshot of Turbinia's web interface.

Architectural Updates

As part of these new architectural updates we are consolidating the "cloud" and "local" implementations to be a single stack that is less dependent on managed cloud services and instead uses the same components for both installation types. This means that we will no longer be using PubSub, PSQ, Datastore or Cloud Functions and will only be supporting what was previously called the "local" stack equivalents for both cloud and local installations.  Here is the mapping of the old components to the new components (all of which were previously in use with the exception of the API server).


  • PSQ  → Celery

  • Datastore → Redis

  • PubSub → Kombu

  • Cloud Functions → API Server


Consolidating these services has several benefits including reduced maintenance and upkeep of parallel infrastructure components, greatly simplified library dependencies and related maintenance and it will also be much simpler to port to new cloud providers (stay tuned!).  External integrations are now also much easier with the API server and shouldn't require separate implementations for the different installation types either.


The last two releases have fully supported the new architecture and after the next release we will begin to remove the old components.  If you have any questions about migrating old integrations or how to integrate new things using this architecture, please file an issue or ask on the Turbinia channel on the OSDFIR Slack.

Summary

The new Turbinia API server and web interface provide a number of new features and improvements that make it easier to use Turbinia. The API server allows users to submit custom requests, retrieve the status of requests and tasks, and download output files. The web interface provides a user-friendly way to monitor tasks, view results, and track them in real-time.


We hope you enjoy using the new features and functionality! The easiest way to stay up-to-date on future releases is by starring the Turbinia GitHub repository.

Comments

Post a Comment

Popular posts from this blog

Parsing the $MFT NTFS metadata file

By
Background Numerous articles have been written about parsing the $MFT NTFS metadata file before. Some of these explain the technique itself [ 1 ] and others mostly focus on how to use tools. Rarely do articles mention the pros and cons of the technique from a digital forensics perspective [ 2 ].  What is $MFT parsing?  For readers not familiar with the NTFS file system, MFT stands for Master File Table. This table is stored in a file system metadata file, meaning that as far as the file system is concerned it is a file, however that the file contains file system metadata not “regular” content. The name of this file is “$MFT”.  The MFT consists of a sequence of (predetermined) fixed-size entries (or MFT entries). These entries are typically 1024 bytes in size, however the size is defined in the NTFS volume header and in a “regular” MFT entry itself.  The first 4-bytes of a “regular” MFT-entry (or record) starts with the signature “FILE”. The MFT entry can also be filled wit
Read more

Incident Response in the Cloud

By
Many organizations have begun moving a majority of their services towards the cloud in recent years. As a result, attackers have shifted their focus towards the cloud. This has resulted in new techniques and methods specifically designed to compromise cloud infrastructure, like the recent SALTSTACK vulnerability that was widely exploited [ 1 ]. Therefore it is critical for these organizations to have an incident response team that understands the new risks attached to cloud and how cloud can make incident response easier or harder.  In this blog post we will walk you through each phase you may encounter in traditional incident response and highlight the differences when adopting cloud computing. It is aimed towards both those who are new to incident response and cloud computing. We’ve included insights that will benefit organizations taking that step towards cloud who want to ensure they are prepared to respond efficiently to cloud incidents.  Traditional Incident Response Before jump
Read more

Container Forensics with Docker Explorer

By Anonymous
  Introduction As previous blog posts on cloud forensics have noted, applications are increasingly being deployed using container orchestration frameworks such as Kubernetes, especially in cloud environments.  Similar to traditional deployments on physical servers or virtual machines (VMs), when a containerised application has a security issue it can lead to a compromise of the underlying compute architecture. In the case of container deployments this means a compromise of the container itself, the container host, or even a wider cluster compromise via abuse of orchestration tools. Often digital forensics is required to establish what went wrong and remediate any issues. This article will provide an introduction to container forensics with Docker Explorer by working through a scenario involving a compromised container running within a Kubernetes cluster. Although Kubernetes is briefly mentioned, this article will focus on analysis of an individual container rather than the wider clust
Read more