log2timeline filtering 101.

By Anonymous

Filtering

The new version of log2timeline introduced lot of new features and changes, one being the new filtering capabilities of the tool.

Let's explore that capability a bit more in this post. First of all the tool is capable of filtering against every stored attribute inside each stored event. To be able to properly know how to use the filters we therefore need to know a bit about how those events get stored. There is always the possibility to just read the definition of an EventObject protobuf, which is the mechanism in which the event is stored in.

However there are other ways we can explore the structure. One would involve using the plaso_console and play with the entries (let's save that for future blog posts) or simply dump the data in it's raw format (as in just print the protobufs). In this post we will be exploring that option, simply printing out the data in raw format.

Let's start with creating a small storage file using the test_data provided with plaso.
log2timeline.py -w /tmp/reg.dump test_data/NTUSER.DAT
[INFO] (MainProcess) Starting storage thread.
[INFO] (MainProcess) Starting to collect files for processing.
[INFO] (MainProcess) Starting to extract events.
[INFO] (MainProcess) Collection is hereby DONE
[INFO] (MainProcess) Waiting until all processing is done.
[INFO] (MainProcess) Processing done, waiting for storage.
[INFO] (StorageThread) [Storage] Closing the storage, nr. of
events processed: 1145
[INFO] (MainProcess) Storage process is done.
[INFO] (MainProcess) Run completed. 
Now we can start looking into the file a bit, let's print out just the first few entries, just to get a feeling of how the entries are stored there:
psort.py -o Raw /tmp/reg.dump "date > 0"  | head -40 
timestamp: 1249398682811068
timestamp_desc: "Last Written"
source_short: REG
source_long: "NTUSER key"
description_long: "UEME_RUNPIDL:%csidl2%\\MSN.lnk: [Count: 14]"
description_short: "[UEME_RUNPIDL:%csidl2%\\MSN.lnk: [Count: 14]]"
attributes {
  key: "plugin"
  value: "XPUserAssistPlugin"
}
attributes {
  key: "parser"
  value: "WinRegistry"
}
attributes {
  key: "inode"
  value: "44230888"
}
filename: "test_data/NTUSER.DAT"
display_name: "test_data/NTUSER.DAT"
pathspec {
  type: OS
  file_path: "test_data/NTUSER.DAT"
}
offset: 350056
store_number: 1
store_index: 4
...

Here we can see a single event and all the attributes that are stored within it. To produce this I used the very simple filter of "date > 0", which just indicates that the we are searching for entries that have a timestamp. By defining an attribute I automatically filter out all events that don't have that attribute set and I also indicate I want that attribute to be of higher value than zero, so it has a defined timestamp.

I highlighted all the attributes in this EventObject, but one thing you need to realize is that some of these attributes will be shared among all EventObjects (think source_long, source_short, description_long, etc.) while others are specifically set by individual parsers, like the "plugin" attribute which is only set by the registry parser. Most of the attributes that are stored inside the "attributes" entries inside the event are parser specific, so they will vary (some of them are set by the tool itself and thus will be consistent).

Now that we know the attributes that are stored there we can start filtering for some of those. For instance:
psort.py /tmp/reg.dump "plugin is 'XPUserAssistPlugin'"
date,time,timezone,MACB,source,sourcetype,type,user,host,short,
desc,version,filename,inode,notes,format,extra
01/01/1970,00:00:00,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_CTLCUACount:ctor: [Count: 2]],UEME_CTLCUACount:ctor: [Count: 2],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:11:22,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPIDL:%csidl2%\MSN.lnk: [Count: 14]],UEME_RUNPIDL:%csidl2%\MSN.lnk: [Count: 14],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:11:22,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPIDL:%csidl2%\Windows Messenger.lnk: [Count: 12]],UEME_RUNPIDL:%csidl2%\Windows Messenger.lnk: [Count: 12],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:11:22,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPIDL:%csidl2%\Windows Media Player.lnk: [Count: 13]],UEME_RUNPIDL:%csidl2%\Windows Media Player.lnk: [Count: 13],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:11:22,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPIDL:%csidl2%\Accessories\Tour Windows XP.lnk: [Count: 11]],UEME_RUNPIDL:%csidl2%\Accessories\Tour Windows XP.lnk: [Count: 11],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:11:22,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPIDL:%csidl2%\Accessories\System Tools\Files and Settings Transfer Wi...],UEME_RUNPIDL:%csidl2%\Accessories\System Tools\Files and Settings Transfer Wizard.lnk: [Count: 10],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:13:42,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPIDL:::{2559A1F4-21D7-11D4-BDAF-00C04F60B9F0}: [Count: 1]],UEME_RUNPIDL:::{2559A1F4-21D7-11D4-BDAF-00C04F60B9F0}: [Count: 1],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:13:42,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPATH:C:\Program Files\Internet Explorer\iexplore.exe: [Count: 1]],UEME_RUNPATH:C:\Program Files\Internet Explorer\iexplore.exe: [Count: 1],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:19:23,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPATH:C:\WINDOWS\system32\NOTEPAD.EXE: [Count: 2]],UEME_RUNPATH:C:\WINDOWS\system32\NOTEPAD.EXE: [Count: 2],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:21:52,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPIDL:%csidl2%\BCWipe 3.0\BCWipe Task Manager.lnk: [Count: 1]],UEME_RUNPIDL:%csidl2%\BCWipe 3.0\BCWipe Task Manager.lnk: [Count: 1],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:21:52,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPIDL: [Count: 3]],UEME_RUNPIDL: [Count: 3],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:21:52,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPIDL:%csidl2%\BCWipe 3.0: [Count: 1]],UEME_RUNPIDL:%csidl2%\BCWipe 3.0: [Count: 1],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:21:53,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPATH: [Count: 4]],UEME_RUNPATH: [Count: 4],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:21:53,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPATH:C:\Program Files\Jetico\BCWipe\BCWipeTM.exe: [Count: 1]],UEME_RUNPATH:C:\Program Files\Jetico\BCWipe\BCWipeTM.exe: [Count: 1],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
[INFO] Events filtered out: 1131
This filter simply indicates that an attribute called "plugin" should exist and its value should equal to "XPUserAssistPlugin", otherwise the filter will skip that entry. This will also automatically filter out all events that do not have the attribute plugin set.

We might want to narrow down the time range:
psort.py /tmp/reg.dump "plugin is 'XPUserAssistPlugin' and date > '2009-08-04 15:15:00'"
date,time,timezone,MACB,source,sourcetype,type,user,host,short,
desc,version,filename,inode,notes,format,extra
08/04/2009,15:19:23,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPATH:C:\WINDOWS\system32\NOTEPAD.EXE: [Count: 2]],UEME_RUNPATH:C:\WINDOWS\system32\NOTEPAD.EXE: [Count: 2],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:21:52,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPIDL:%csidl2%\BCWipe 3.0\BCWipe Task Manager.lnk: [Count: 1]],UEME_RUNPIDL:%csidl2%\BCWipe 3.0\BCWipe Task Manager.lnk: [Count: 1],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:21:52,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPIDL: [Count: 3]],UEME_RUNPIDL: [Count: 3],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:21:52,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPIDL:%csidl2%\BCWipe 3.0: [Count: 1]],UEME_RUNPIDL:%csidl2%\BCWipe 3.0: [Count: 1],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:21:53,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPATH: [Count: 4]],UEME_RUNPATH: [Count: 4],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
08/04/2009,15:21:53,UTC,..C.,REG,NTUSER key,Last Written,-,-,[UEME_RUNPATH:C:\Program Files\Jetico\BCWipe\BCWipeTM.exe: [Count: 1]],UEME_RUNPATH:C:\Program Files\Jetico\BCWipe\BCWipeTM.exe: [Count: 1],2,test_data/NTUSER.DAT,44230888,-,WinRegistry,plugin: XPUserAssistPlugin
[INFO] Events filtered out: 1139

Let's brake down this filter query:
plugin is 'XPUserAssistPlugin' and date > '2009-08-04 15:15:00'"
Again we are filtering out all events that do not have the attribute plugin set and we are narrowing it down to only those events that have the plugin attribute set to "XPUserAssistPlugin", but we are also narrowing down the date. The "date" keyword is a special keyword that gets translated into "timestamp", which is the actual attribute stored in the EventObject (noticed there is no attribute called date there). And the date attribute accepts "human readable" dates which get transformed into an integer that is later used for the comparison. So now we can filter and say that we only want events that both have the plugin attribute set to "XPUserAssistPlugin" AND occur after 4th of August 2009 15:15:00.

Finally let's put together a slightly more complex filter, we know that we are looking for an entry in the UserAssist key that has something to do with wipe in the word and it is either an executable or a shortcut we are pointing to.

psort.py /tmp/reg.dump "plugin contains 'userassist' and (description_long iregexp 'wipe.+exe' or description_long iregexp 'wipe.+lnk')" | awk -F ',' '{print $1,$2,$11}'
[INFO] Events filtered out: 1143
date time desc
08/04/2009 15:21:52 UEME_RUNPIDL:%csidl2%\BCWipe 3.0\BCWipe Task Manager.lnk: [Count: 1]
08/04/2009 15:21:53 UEME_RUNPATH:C:\Program Files\Jetico\BCWipe\BCWipeTM.exe: [Count: 1]

Let's brake down the filter one more time:
"plugin contains 'userassist' and (description_long iregexp 'wipe.+exe' or description_long iregexp 'wipe.+lnk')"
Here we really have two conditions, the plugin attribute has to exist and contain (case insensitive) the word userassist in it (catching all UserAssist plugins). And the description_long attribute has to contain the regular expression pattern "wipe.+exe" or "wipe.+lnk". Since we use the keyword iregexp this regular expression is compiled as case insensitive (regex is case sensitive).

In other words we are simply saying that we are looking for an event that has the plugin attribute set to contain the word "userassist" and the description_long attribute contains either the pattern "wipe.+exe" or "wipe.+lnk" (case insensitive). The last part of the command was a simple awk command indicating that we would only like to print out the date, time and desc fields of the l2t_csv format (default output of psort).

Simply put the filters in log2timeline are really powerful and can be used to match against any attribute set in the event and combined to make finding those needles easier (given that you know what you are looking for).

And as an added bonus these filters can also be applied to log2timeline to filter out events BEFORE they are stored on disk. Try the same filter as before only this time apply it to the processing:

log2timeline.py -w /tmp/reg2.dump test_data/NTUSER.DAT "plugin contains 'userassist' and (description_long iregexp 'wipe.+exe' or description_long iregexp 'wipe.+lnk')"
[INFO] (MainProcess) Starting storage thread.
[INFO] (MainProcess) Starting to collect files for processing.
[INFO] (MainProcess) Starting to extract events.
[INFO] (MainProcess) Collection is hereby DONE
[INFO] (MainProcess) Waiting until all processing is done.
[INFO] (MainProcess) Processing done, waiting for storage.
[INFO] (StorageThread) [Storage] Closing the storage, nr. of events processed: 2
[INFO] (MainProcess) Storage process is done.
[INFO] (MainProcess) Run completed. 
psort.py /tmp/reg2.dump | awk -F ',' '{print $11}'desc
UEME_RUNPIDL:%csidl2%\BCWipe 3.0\BCWipe Task Manager.lnk: [Count: 1]
UEME_RUNPATH:C:\Program Files\Jetico\BCWipe\BCWipeTM.exe: [Count: 1]
Now we see that the storage file only contains those two events, no other events were stored on disk.

Comments

Post a Comment

Popular posts from this blog

Parsing the $MFT NTFS metadata file

By
Background Numerous articles have been written about parsing the $MFT NTFS metadata file before. Some of these explain the technique itself [ 1 ] and others mostly focus on how to use tools. Rarely do articles mention the pros and cons of the technique from a digital forensics perspective [ 2 ].  What is $MFT parsing?  For readers not familiar with the NTFS file system, MFT stands for Master File Table. This table is stored in a file system metadata file, meaning that as far as the file system is concerned it is a file, however that the file contains file system metadata not “regular” content. The name of this file is “$MFT”.  The MFT consists of a sequence of (predetermined) fixed-size entries (or MFT entries). These entries are typically 1024 bytes in size, however the size is defined in the NTFS volume header and in a “regular” MFT entry itself.  The first 4-bytes of a “regular” MFT-entry (or record) starts with the signature “FILE”. The MFT entry can...
Read more

Incident Response in the Cloud

By
Many organizations have begun moving a majority of their services towards the cloud in recent years. As a result, attackers have shifted their focus towards the cloud. This has resulted in new techniques and methods specifically designed to compromise cloud infrastructure, like the recent SALTSTACK vulnerability that was widely exploited [ 1 ]. Therefore it is critical for these organizations to have an incident response team that understands the new risks attached to cloud and how cloud can make incident response easier or harder.  In this blog post we will walk you through each phase you may encounter in traditional incident response and highlight the differences when adopting cloud computing. It is aimed towards both those who are new to incident response and cloud computing. We’ve included insights that will benefit organizations taking that step towards cloud who want to ensure they are prepared to respond efficiently to cloud incidents.  Traditional Incident Response Be...
Read more

Container Forensics with Docker Explorer

By Anonymous
  Introduction As previous blog posts on cloud forensics have noted, applications are increasingly being deployed using container orchestration frameworks such as Kubernetes, especially in cloud environments.  Similar to traditional deployments on physical servers or virtual machines (VMs), when a containerised application has a security issue it can lead to a compromise of the underlying compute architecture. In the case of container deployments this means a compromise of the container itself, the container host, or even a wider cluster compromise via abuse of orchestration tools. Often digital forensics is required to establish what went wrong and remediate any issues. This article will provide an introduction to container forensics with Docker Explorer by working through a scenario involving a compromised container running within a Kubernetes cluster. Although Kubernetes is briefly mentioned, this article will focus on analysis of an individual container rather than the wi...
Read more