An update on the latest Turbinia features

The Turbinia development team are excited to announce the release of a number of new features and improvements for Turbinia. With the latest updates, Turbinia now includes an API server and a web interface, providing even more flexibility and ease of use. In this blog post we'll take a closer look at these exciting new additions.


Turbinia API Server

Overview

The Turbinia API server is an HTTP REST API that allows clients to submit forensic processing requests and monitor their progress. This opens up new possibilities for integrating Turbinia into other applications or systems, automating forensic processing, and streamlining investigations. 

dfTimewolf–a forensics workflow orchestration tool– is now integrated with Turbinia's API server to create new evidence processing flows.

Features

The API server provides several endpoints to perform different actions. At time of writing this article, the API server supports the following actions:

  • Create custom Turbinia requests (e.g. using recipes or custom POST data parameters)

  • Retrieve the status of requests and their associated tasks to track their progress

  • Retrieve server configuration settings

  • Retrieve a list of enabled Turbinia job types

  • Download request and task output files, such as Plaso storage files

  • Download worker and task log files for easier troubleshooting

We built the API server using OpenAPI as a standard and FastAPI as a framework. It is designed to be flexible and extensible, which allows you to define your own forensic tasks as JSON documents, and more easily integrate Turbinia into existing workflows or systems. Until now, the only ways to integrate Turbinia with other tools was to import a Python client directly in code, or turbiniactl cli tool. The API server and client library make integration easier through HTTP requests, and a lightweight Python package without being required to install all of Turbinia's components and dependencies.

To protect sensitive endpoints and forensic data, the server supports authentication via OAuth2 or OpenID Connect (OIDC) identity providers through oauth2-proxy. This allows for single sign-on through different identity providers like Google, AWS, Okta, etc.

API documentation and a web-based client is available at /docs. The docs UI can be used to test and interact with the different endpoints.

Figure 1. Turbinia documentation and REST client

Interacting with the API

In addition to the built-in web-based client, we created two new packages which are available via PyPi (a Python package management platform accessible via the pip command). A new Python library turbinia-api-lib that can be used to interact with the API server programmatically, and a new command-line tool distributed as turbinia-client which provides a convenient way to interact with the Turbinia API server from the command line. 

The turbinia-client package offers a range of functionality that enables you to submit tasks, monitor their progress and retrieve available results. User documentation and usage examples can be found here.

Usage examples

Available commands can be listed by running the following command:

turbinia-client -h


Options:

  -c, --config_instance TEXT  A Turbinia instance configuration name.

                              [default: (dynamic)]

  -p, --config_path TEXT      Path to the .turbinia_api_config.json file..

                              [default: (dynamic)]

  -h, --help                  Show this message and exit.


Commands:

  config  Get Turbinia configuration.

  jobs    Get a list of enabled Turbinia jobs.

  result  Get Turbinia task or request results.

  status  Get Turbinia request/task status.

  submit  Submit new requests to the Turbinia API server.


To retrieve the status of an existing Turbinia request we would invoke the status command, and the request sub-command as follows:

turbinia-client status request -j 65c873083075409faa6c70f5bb30c6b6

The -j option indicates we want JSON output instead of the default Markdown status report.

{

  "failed_tasks": 0,

  "last_task_update_time": "2023-05-10T21:42:36.788874Z",

  "queued_tasks": 0,

  "reason": "case123",

  "request_id": "65c873083075409faa6c70f5bb30c6b6",

  "requester": "user1",

  "running_tasks": 0,

  "status": "successful",

  "successful_tasks": 34,

"task_count": 34,

  "tasks": [

    {

      "all_args": "",

      "evidence_name": "GoogleCloudDisk:/p15",

      "group_id": "86580b15215241b89ddb3d07ab52bffb",

      "group_name": "",

      "id": "e1d92289f830486c9bce76cd72643fbe",

      "instance": "osdfir-cluster",

      "job_id": "828e6f77c78841329e05ff45ee4ce398",

      "last_update": "2023-05-10T20:32:38.757347Z",

      "name": "ContainerdEnumerationTask",

      "reason": "case123",

      "report_data": null,

      "report_priority": 50,

      "request_id": "65c873083075409faa6c70f5bb30c6b6",

      "requester": "user1",

      "run_time": 0.051558,

      "saved_paths": [

        "/mnt/turbiniavolume/output/65c873083075409faa6c70f5bb30c6b6/1683750749-e1d92289f830486c9bce76cd72643fbe-ContainerdEnumerationTask/worker-log.txt"

      ],

      "status": "Found 0 containers",

      "successful": true,

      "worker_name": "turbinia-worker-7ddc986664-pcrzg"

    },

Turbinia Web UI

We are also excited to release a top requested feature, a new web interface for Turbinia. The web interface provides a user-friendly way of interacting with Turbinia's new features. It offers a visual representation of the processing tasks, their progress, and results in a web-based dashboard. This makes it easy for investigators to monitor forensic processing tasks, view results, and track them in real-time.

Figure 2. Screenshot of Turbinia's web interface.

Architectural Updates

As part of these new architectural updates we are consolidating the "cloud" and "local" implementations to be a single stack that is less dependent on managed cloud services and instead uses the same components for both installation types. This means that we will no longer be using PubSub, PSQ, Datastore or Cloud Functions and will only be supporting what was previously called the "local" stack equivalents for both cloud and local installations.  Here is the mapping of the old components to the new components (all of which were previously in use with the exception of the API server).


  • PSQ  → Celery

  • Datastore → Redis

  • PubSub → Kombu

  • Cloud Functions → API Server


Consolidating these services has several benefits including reduced maintenance and upkeep of parallel infrastructure components, greatly simplified library dependencies and related maintenance and it will also be much simpler to port to new cloud providers (stay tuned!).  External integrations are now also much easier with the API server and shouldn't require separate implementations for the different installation types either.


The last two releases have fully supported the new architecture and after the next release we will begin to remove the old components.  If you have any questions about migrating old integrations or how to integrate new things using this architecture, please file an issue or ask on the Turbinia channel on the OSDFIR Slack.

Summary

The new Turbinia API server and web interface provide a number of new features and improvements that make it easier to use Turbinia. The API server allows users to submit custom requests, retrieve the status of requests and tasks, and download output files. The web interface provides a user-friendly way to monitor tasks, view results, and track them in real-time.


We hope you enjoy using the new features and functionality! The easiest way to stay up-to-date on future releases is by starring the Turbinia GitHub repository.


Comments

Popular posts from this blog

Parsing the $MFT NTFS metadata file

Incident Response in the Cloud

Container Forensics with Docker Explorer