Visualize the Output

log2timeline has always been pretty good at aggregating quite a hefty load of data, making it easy to crash spreadsheet applications like Microsoft Excel. The goal was usually to collect everything that may be of value and present that to whoever uses the tool. Yet when it came to the actual analysis of the resulting data set that has always been left as an exercise to the reader. And for some that might have proved to be quite a burden.

Well, this is obviously all changing now with the introduction of plaso.... We've been trying to improve this experience lately, with the introduction of time slices (discussed in a future blog post), tagging and filtering. That is we added the option of tagging the resulting output from log2timeline with annotations like "Application Executed", "Document Printed", etc. You can then filter the data set and only display the tagged entries and even include surrounding events using time slices. This makes drilling down into the data set quite quick and easy.

However we wanted to do more, and what better way of exploring the data set than to load it up in some sort of database and visualize the output. Visualization is getting all the buzz and we definitely didn't want to be left behind... so why not use something that's already out there instead of re-inventing the wheel?

Let me introduce you to Kibana and ElasticSearch. There is even a demo of Kibana out there that you can play with just to see what can be done with it.

But how does all of this relate to plaso and log2timeline? What can be done to combine these two products? Well....

Installation

ElasticSearch

First of all we need to install ElasticSearch. Go to the download page and get the package that is suitable for your platform. Let's assume we downloaded the .deb package and we are running this on an Ubuntu system.

sudo dpkg -i elasticsearch-0.90.6.deb

And that's it, now elasticsearch is installed. Let's start the service:

sudo /etc/init.d/elasticsearch restart

You might need to install Java to get this to work, for that you need:

sudo apt-get install openjdk-7-jre-headless 

PyElasticSearch

Now ElasticSearch should be installed and all working. But for plaso to pick up the module we need to install the pyelasticsearch module.

git clone https://github.com/rhec/pyelasticsearch.git

cd pyelasticsearch

python setup.py build

sudo python setup.py install

To make sure everything is working we can now do:

psort.py -o list

If things are working we should see the following results:

******************************** Output Modules ********************************
    L2tcsv : The CSV format used by log2timeline, with 17 fixed fields.
   Elastic : Saves the events into an ElasticSearch database.
  Mysql4n6 : Contains functions for outputing as 4n6time MySQL database.
   Dynamic : Dynamic selection of fields for a separated value output format.
     Rawpy : Prints out a "raw" interpretation of the EventObject.
       Raw : Prints out a "raw" interpretation of the EventObject protobuf.
    Sql4n6 : Saves the data in a SQLite database, used by the tool 4n6Time.
  Pstorage : Dumps event objects to a plaso storage file.
--------------------------------------------------------------------------------

Now we can save our output from plaso directly to a local instance of ElasticSearch (later revisions will allow us to save to a database set up on another server).

N.b. this will unfortunately not work with the newly released version of plaso, this requires downloading the latest checked in source via git and installing the tool from source, see instructions here.

Kibana

So Kibana is the visual layout that reads the data from the ElasticSearch database and presents that to the user. Get it here and follow the instructions.

Essentially this requires you to setup a webserver, and configure that to work with Kibana. This can be easily achieved using something like ngix:

sudo apt-get install nginx

Let's untar the kibana installation and install it somewhere we can access it.

tar xfvz kibana-3.0.0milestone4.tar.gz

mv kibana-3.0.0milestone4 /usr/share/kibana3/

Then to configure Kibana, edit the config.js file using your favorite text editor...

vim /usr/share/kibana3/config.js

And change the "elasticsearch" variable to point to your host.

And then we need to configure nginx. This can be done following the simple sample configuration file from Kibana, accessible here.

Example configuration:

wget https://raw.github.com/elasticsearch/kibana/master/sample/nginx.conf 

sudo mv nginx.conf /etc/nginx/sites-available/kibana
cd /etc/nginx/sites-enabled
rm default
ln -s ../sites-available/kibana .

Edit the kibana file using your favorite text editor to make necessary adjustments from the sample config file. You really want to change the "server_name" variable to reflect your hostname, and potentially password protect your site.

Necessary Disclaimer

All this setup essentially setups ElasticSearch database and a webserver running Kibana on your local workstation, open up to everyone that has access to your box. You may want to not follow these simple instructions and do this properly so that access to your information is locked and not available to everyone that has network access to your computer. This is just to demonstrate how this can be achieved simply, not securely.... so don't blame me when your data is exposed to everyone on your network. And there is also few other steps that I purposly skipped to make this blog post a bit shorter.

Putting It All Together

OK, so now we should have a working setup of ElasticSearch and Nginx running the Kabana site, and now we can start playing with the database. First things first, we need to include some data there. Let's fire up psort and include data in the timeline:

psort.py -o elastic nfury.dump

Here I'm including the data set extracted from the Windows 7 64-bit image provided by SANS (the 508 class). And we can also include the results from nromanoff's image from SANS 508.

psort.py -o elastic nromanoff.dump

And this is it... that's all that needs to be done. Now all the data is entered into the database and now all we need to do is to open up a browser and look at the resulting data. And we even have the data correlated between these two disks, in the same view.

The data set can now be explored, filtered and pretty graphs displayed. An example dashboard can be downloaded from here: https://plaso.googlecode.com/git/extra/plaso_kibana_example.json

Just save this file under:

/usr/share/kibana3/app/dashboards/plaso.json

And it should be accessible as "Plaso" in the "Load" button

If not, you can always enter it by browsing to this site:

http://yourhostname/index.html#/dashboard/file/plaso.json

You can then enter filters, select which fields should be visible and interact with the histogram to zoom into a particular date range.

An example screenshot can be found here:

And to inspect additional details about a particular event, just click it to expand it:

And within that view you can exclude this particular event from view, or to filter for other like that.

Here is an example screenshot taken from entering the query "Application Execution" into the query field and selecting an area in the time histogram that resembled a recent peak in activity.

And the results: