Case study of comparing command line tools
Case study of comparing command line tools
Background
Recently I ran into an issue where one of the tools I was using no longer produced the output I was expecting. I had used the tool just a couple of days ago where it provided me the desired results. The source code of the tool had not changed. I had changed the Linux distribution on which the tool was built and running. After some debugging I concluded that the different distribution enabled the compiler flag “-faggressive-loop-optimizations” by default. Compilation did not fail and unfortunately this flag removed an important loop in the code altering the behavior of the tool.
This got me thinking what other alterations could the optimizer have made? How would I tell? How would I catch this at scale? What if my distro had built the binary, would they have caught the issue?
In “Forensics tools” where are your tests? I emphasized that “The absence of documented tests and results renders reproducibility impossible.” Unfortunately in digital forensics and incident response (sometimes referred to as DFIR) it is very common to claim something is a “digital forensics tool” without backing this with any documented proof. First, this is doing a disservice to people new to the field trying to learn digital forensics or practitioners that have to deal with shortcomings in these tools in a legal setting where tolerance for error is low. Second, it also means as a field we are unable to objectively validate and reason about expected tool behavior.
One could argue that it is tricky (or costly) given such a baseline would need to be updated frequently, but without it, we can only speculate about the output of a tool.
Sometimes the advice is given to use different tools to validate findings but unfortunately this is not a foolproof method either, sometimes our understanding of corresponding data formats is of low maturity or tool authors are known to adopt erroneous practices from other tools. Also see: Testing digital forensic data processing tools and https://github.com/log2timeline/dfvfs/issues/230.
Comparing apples and oranges
However for most software there are one or more reference implementations. For example for the Extended File System (ext) one could argue that the Linux kernel is the reference implementation and that a tool like (coreutils) stat could be used to query a running kernel and establish some sort of a baseline.
Note that using a complex piece of software like an operating system can be tricky. Various parts might influence our observation. Hence let’s also consider using debugfs as an additional reference implementation closer to the data.
Coreutils stat (http://git.savannah.gnu.org/gitweb/?p=coreutils.git), reference implementation that uses the Linux kernel;
E2fsprogs debugfs (https://git.kernel.org/pub/scm/fs/ext2/e2fsprogs.git), reference implementation from the same source as the Linux kernel but closer to the data;
Libfsext fsextinfo (https://github.com/libyal/libfsext), implementation of ext file system parser the author of this article knows was done independently;
SleutKit istat (https://github.com/sleuthkit/sleuthkit), collection of command line digital forensics tools that allow you to investigate volume and file system data.
This gives us 4 different implementations of tooling that could “stat” information from an ext inode. Versions of the tooling used:
As distributed by Fedora 44
As distributed by Fedora 44
Built from commit 3d75aad11d037a8a583d9dbb76f2c9692f898878 on Fedora 44
Built from commit 4936f765d36d523d52a1925fa2f3ed11a5fcae65 on Fedora 44 without optimization (due to #3498).
Test data was generated using the ext-specimens scripts with mke2fs 1.47.0 on Ubuntu 26.04. Note that this version defaults to an inode size of 256.
What is in a stat?
To be able to set a baseline it would be good to determine what information needs to go into it. Let’s determine what is in a stat?
Based on “man 3 stat” on Fedora 44, struct stat contains:
Based on “man 2 statx” on Fedora 44, struct statx contains:
One could conclude a (baseline) stat consists of:
An inode number, though technically not stored within an ext inode
File type and mode
Number of hard links
User and group identifier of owner
Optional device identifier
File size, in bytes and in number of 512 byte blocks
Time of last access, last modification, last status change (or inode change time)
Optional time of creation
Note that the “ID of device containing file” is a runtime value and would only be available if the file system is mounted; and “Block size for filesystem I/O” applies to the entire file system and is not inode specific. Also one could argue including both the file size in bytes and number of 512 byte blocks is redundant.
This highlights an important aspect, who determines what is included in the baseline? And what not? And more importantly why?
Let us also assess if some existing onthologies can provide us answers to these questions to improve our makeshift baseline of stat.
The CASE ontology defines ExtInodeFacet, which includes deletion time, where struct stat and struct statx do not. CASE also defines several comparable observables such as FileSystemObject. Alternatively the closest to a struct stat in STIX appears to be the file observable.
Unfortunately neither of 2 ontologies appear to include a rationale about why values are included or how the observables compare to other implementations such as struct stat or struct statx.
Since solving this nomenclature problem is not the issue we wanted to address, let’s stick with our makeshift definition of a “stat observable” and next look at the output of the tools on both ext2 and ext4 file systems.
Stat of ext2
Observations based on the output and our makeshift definition of “stat observable”:
Stat and istat do not include tool name and version in their output. Debugfs prints tool name and version to stderr.
Date and time values in the output of stat, debugfs and istat are dependent on the TZ environment variable.
Debugfs includes no time zone information and istat uses a potential ambiguous time zone representation (when run with a different TZ), for context see time zones and daylight savings.
Debugfs and fsextinfo use a (potential) localized date format, such as name of month, where stat and istat use an ISO 8601-like format.
Several of the tools include additional values such as “flags” or “group” or exclude values like “block size”.
Debugfs presents time values in seconds granularity but does include low-level timestamp information, such as “0x6a0df2fc:d549c330”. Istat presents inode times values in seconds granularity but does not include additional timestamp information.
“File Created” is not included in the istat output.
Extended attributes are not included in the stat and istat output.
Stat of ext4
Observations:
In contrast to the ext2 stat output istat now includes “flags” and “File Created” in its output
Istat output hints at use of “Direct Blocks” in contrast to debugfs which correctly indicates the use of extents
Normalization and comparison
As you can see from both examples, even in such a confined problem space, there is already significant variation in output. The output of the tools is comparable for a human but not sufficiently similar for automated (deterministic) comparison. Based on our reference implementations and makeshift definition of “stat observable” let’s create a normalized output of for inode number 22 in the ext2.raw test file:
Let’s consider some basic rules for comparison:
The tool output must include all attributes defined in the baseline;
The normalized value of an attribute (from the tool output) must match that in the baseline.
The second comparison rule introduced an additional challenge. Setting TZ=UTC helped us with not having to deal with time zone and daylight savings conversions, but as you might recall istat only provided the date and time values in seconds granularity. A strict comparison would require the granularity to match the granularity of the baseline, given this mismatch of granularity can lead to analysis errors. Also see pearls and pitfalls of timeline analysis.
From a validation perspective we likely want to treat such variation differently from a value mismatch, e.g. by indicating the match is comparable but not exact (comparable to a round-off error).
The “stat observable” introduces another interesting edge case, which did not surface in the examples. A stat of a directory will contain the size of the directory metadata. A tool might decide that the size of a directory is conceptually different and therefore set size to 0 for a directory.
To still be able to compare that tool against other tools we might require more specific rules such as to skip the size of a directory if set to 0. From a transparency perspective we want to document such behavior.
Meet clitooltester
To perform tool comparison at scale we can use clitooltester. First we need test data, in this article we used ext-specimens. We want this test data to be representative for many different variations (or cases) of the data format, such as different versions of the format (ext2, ext3, ext3), different inode sizes, and so forth.
Next we need to define our test case (inputs) and reference data. For example for inode 22 in ext2.raw example we expect the output to contain:
It could be beneficial to document this baseline. When processing actual case data we can extract these traits and check if our baseline covers them and we can selectively apply additional scrutiny to newly observed data format variations, potentially strengthening our findings.
Once we have the test data, inputs and reference data we can run clitooltester against an installation. For the sake of simplicity let’s consider a Docker image as our installation and create a clitooltester test configuration for istat, which we name “istat.yaml”.
And configuration of test inputs, which we name “inputs.yaml”:
If we now run the clitooltester run.py script it will:
Build the “installation”, as a Docker image with tag “pytsk_tests/sleuthkit”, based on the Docker file, “sleuthkit/Dockerfile”
Run istat based on the command within a Docker container using the Docker image with tag “pytsk_tests/sleuthkit”
Capture stdout of istat and normalize it using “sleuthkit/istat.py”
Compare the normalized istat output against an input specific reference file using “scripts/json_validator.py”
Note that clitooltester is an early alpha release, so there are many improvements that can be made, such as capturing the reason for the test failing in a log file. For now the -v option provides these details:
This corresponds with what we observed ourselves:
the output contained information about allocated, group and NFS generation number that is not in our reference data;
creation time is missing from the output;
the granularity of date and time values is different from our reference data.
The latest version of the tests can be found at https://github.com/py4n6/pytsk_tests.
Conclusion
The approach outlined in this article allows us (as tool users) to at least create a baseline level of confidence in the tools we use. Unfortunately creating these baselines can be tricky and time consuming.
Several aspects of creating such baselines we deliberately skipped over in this article such as:
What is the ontology of our baseline? What information do observables and values represent? What values are part of an observable and values are not? And why?
How do observables relate to the original digital information? E.g. how does CASE ExtInodeFacet correspond to struct stat?
Was something a runtime only information such as the “ID of device containing file” in struct stat or was it stored as part of a specific data structure?
Are there authoritative sources for such a baseline? If so, where are they and what makes them authoritative?
What if there are multiple reference implementations, e.g. for the FAT file system?
What is the expected (or desired) behavior of data format (edge) cases not supported by the reference implementation?
We also surfaced several reflective questions for practitioners and tool authors:
Should a tool include its name and version in the output? Should such a header include more metadata than the name of the tool and version?
Should a tool support localization or does this only lead to potential interpretation errors?
Although a critical part of digital forensics systematic tool testing and validation appears to not be broadly discussed or applied within the field. If you have ideas to progress the field in this topic don’t hesitate to reach out via the Open Source DFIR Slack community.
A final note on the advice to use different or dual tools to validate finding is a loaded one. In theory this might sound like a good way to validate tools however in practice:
You might be comparing different concepts (apples and oranges);
The tools might not be as independent as you need them to be; different authors making the same mistakes or worse authors copying erroneous logic;
As an industry, we might have limited understanding of the data format.
In addition the following may be an interesting read on the topic: Reliability validation for file system interpretation.
Remember you as the digital forensics analyst need to understand the technique, potential edge cases and how your tooling deals with these. You are the one that potentially needs to defend them in court (or other legal setting), not a tool vendor, book author, a teacher/instructor, a researcher or an opinionated peer in the field.
Comments
Post a Comment