An update on the latest Turbinia features
The Turbinia development team are excited to announce the release of a number of new features and improvements for Turbinia. With the latest updates, Turbinia now includes an API server and a web interface, providing even more flexibility and ease of use. In this blog post we'll take a closer look at these exciting new additions.
Turbinia API Server
Overview
The Turbinia API server is an HTTP REST API that allows clients to submit forensic processing requests and monitor their progress. This opens up new possibilities for integrating Turbinia into other applications or systems, automating forensic processing, and streamlining investigations.
dfTimewolf–a forensics workflow orchestration tool– is now integrated with Turbinia's API server to create new evidence processing flows.
Features
The API server provides several endpoints to perform different actions. At time of writing this article, the API server supports the following actions:
Create custom Turbinia requests (e.g. using recipes or custom POST data parameters)
Retrieve the status of requests and their associated tasks to track their progress
Retrieve server configuration settings
Retrieve a list of enabled Turbinia job types
Download request and task output files, such as Plaso storage files
Download worker and task log files for easier troubleshooting
We built the API server using OpenAPI as a standard and FastAPI as a framework. It is designed to be flexible and extensible, which allows you to define your own forensic tasks as JSON documents, and more easily integrate Turbinia into existing workflows or systems. Until now, the only ways to integrate Turbinia with other tools was to import a Python client directly in code, or turbiniactl cli tool. The API server and client library make integration easier through HTTP requests, and a lightweight Python package without being required to install all of Turbinia's components and dependencies.
To protect sensitive endpoints and forensic data, the server supports authentication via OAuth2 or OpenID Connect (OIDC) identity providers through oauth2-proxy. This allows for single sign-on through different identity providers like Google, AWS, Okta, etc.
API documentation and a web-based client is available at /docs. The docs UI can be used to test and interact with the different endpoints.
Figure 1. Turbinia documentation and REST client
Interacting with the API
In addition to the built-in web-based client, we created two new packages which are available via PyPi (a Python package management platform accessible via the pip command). A new Python library turbinia-api-lib that can be used to interact with the API server programmatically, and a new command-line tool distributed as turbinia-client which provides a convenient way to interact with the Turbinia API server from the command line.
The turbinia-client package offers a range of functionality that enables you to submit tasks, monitor their progress and retrieve available results. User documentation and usage examples can be found here.
Usage examples
Available commands can be listed by running the following command:
turbinia-client -h
Options:
-c, --config_instance TEXT A Turbinia instance configuration name.
[default: (dynamic)]
-p, --config_path TEXT Path to the .turbinia_api_config.json file..
[default: (dynamic)]
-h, --help Show this message and exit.
Commands:
config Get Turbinia configuration.
jobs Get a list of enabled Turbinia jobs.
result Get Turbinia task or request results.
status Get Turbinia request/task status.
submit Submit new requests to the Turbinia API server.
To retrieve the status of an existing Turbinia request we would invoke the status command, and the request sub-command as follows:
turbinia-client status request -j 65c873083075409faa6c70f5bb30c6b6
The -j option indicates we want JSON output instead of the default Markdown status report.
{
"failed_tasks": 0,
"last_task_update_time": "2023-05-10T21:42:36.788874Z",
"queued_tasks": 0,
"reason": "case123",
"request_id": "65c873083075409faa6c70f5bb30c6b6",
"requester": "user1",
"running_tasks": 0,
"status": "successful",
"successful_tasks": 34,
"task_count": 34,
"tasks": [
{
"all_args": "",
"evidence_name": "GoogleCloudDisk:/p15",
"group_id": "86580b15215241b89ddb3d07ab52bffb",
"group_name": "",
"id": "e1d92289f830486c9bce76cd72643fbe",
"instance": "osdfir-cluster",
"job_id": "828e6f77c78841329e05ff45ee4ce398",
"last_update": "2023-05-10T20:32:38.757347Z",
"name": "ContainerdEnumerationTask",
"reason": "case123",
"report_data": null,
"report_priority": 50,
"request_id": "65c873083075409faa6c70f5bb30c6b6",
"requester": "user1",
"run_time": 0.051558,
"saved_paths": [
"/mnt/turbiniavolume/output/65c873083075409faa6c70f5bb30c6b6/1683750749-e1d92289f830486c9bce76cd72643fbe-ContainerdEnumerationTask/worker-log.txt"
],
"status": "Found 0 containers",
"successful": true,
"worker_name": "turbinia-worker-7ddc986664-pcrzg"
},
Turbinia Web UI
We are also excited to release a top requested feature, a new web interface for Turbinia. The web interface provides a user-friendly way of interacting with Turbinia's new features. It offers a visual representation of the processing tasks, their progress, and results in a web-based dashboard. This makes it easy for investigators to monitor forensic processing tasks, view results, and track them in real-time.
Figure 2. Screenshot of Turbinia's web interface.
Architectural Updates
As part of these new architectural updates we are consolidating the "cloud" and "local" implementations to be a single stack that is less dependent on managed cloud services and instead uses the same components for both installation types. This means that we will no longer be using PubSub, PSQ, Datastore or Cloud Functions and will only be supporting what was previously called the "local" stack equivalents for both cloud and local installations. Here is the mapping of the old components to the new components (all of which were previously in use with the exception of the API server).
PSQ → Celery
Datastore → Redis
PubSub → Kombu
Cloud Functions → API Server
Consolidating these services has several benefits including reduced maintenance and upkeep of parallel infrastructure components, greatly simplified library dependencies and related maintenance and it will also be much simpler to port to new cloud providers (stay tuned!). External integrations are now also much easier with the API server and shouldn't require separate implementations for the different installation types either.
The last two releases have fully supported the new architecture and after the next release we will begin to remove the old components. If you have any questions about migrating old integrations or how to integrate new things using this architecture, please file an issue or ask on the Turbinia channel on the OSDFIR Slack.
Summary
The new Turbinia API server and web interface provide a number of new features and improvements that make it easier to use Turbinia. The API server allows users to submit custom requests, retrieve the status of requests and tasks, and download output files. The web interface provides a user-friendly way to monitor tasks, view results, and track them in real-time.
We hope you enjoy using the new features and functionality! The easiest way to stay up-to-date on future releases is by starring the Turbinia GitHub repository.
Comments
Post a Comment