Ahoi @roshanmaskey !
First of all: Thank your for your super cool tool! Came in really handy as i am currently working on a master thesis about Kubernetes forensics. :)
I've evaluated your tool for post-mortem forensics by using an exported image from a saved virtual machine which is a worker node in a Kubernetes cluster, where i deployed a pod that pulls a specially crafted container that downloads two more files when active.
When the container runs and DownloadFiles.sh has been executed, the directory /home/SampleFiles/ contains the following files:
# ls -la /home/SampleFiles/
total 8336
drwxr-xr-x 1 root root 4096 Nov 17 17:59 .
drwxr-xr-x 1 root root 4096 Nov 15 23:58 ..
-rwxr-xr-x 1 root root 1089 Nov 15 23:50 DownloadFiles.sh
-rwxr-xr-x 1 root root 2155399 Nov 15 21:09 File01.2MB.tmp
-rwxr-xr-x 1 root root 2097180 Nov 15 21:09 File02.2MB.tmp
-rw-r--r-- 1 root root 2155399 Nov 17 17:59 File03.2MB.tmp
-rw-r--r-- 1 root root 2097180 Nov 17 17:59 File04.2MB.tmp
The used image for the container is: docker.io/pr3l14t0r/forensics:craftedfiles
I've now stumbled across a problem with the implementation of the container mount options.
Context
I can see the contents of that container by manually digging through the snapshotter fs directories by using Autopsy or whatever tool which is capable of parsing the exported disk image.
Now i want to try to mount the container using your tool.
This is how the mount options for the container look like when you run findmnt on the worker node when live:
Target Source FSType Options
------ ------ ------ -------
/run/containerd/io.containerd.runtime.v2.task/k8s.io/2df8c9bf881593eea104ed0cce6234582aef7f26a13fdf034ca1fdd6abf60b46/rootfs overlay overlay rw,relatime,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/44/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/43/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/42/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/41/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/40/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/39/fs,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/45/fs,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/45/work,xino=off
Notice that the commited layers are mounted from last to first, so 44-39 being the lowerdirs and 45 being the upper- and work-dir reflecting the live file system mounted into the container.
So.. the "lowest" lowerdir - the first layer of a container image - needs to be specified last in the lowerdir= parameter.
The Issue
When mounting this container with container-explorer, the following (debug) output is given:
container-explorer --debug --image-root /tmp/investigation -n k8s.io mount 2df8c9bf881593eea104ed0cce6234582aef7f26a13fdf034ca1fdd6abf60b46 /mnt/container
DEBU[0000] user mount command containerid=2df8c9bf881593eea104ed0cce6234582aef7f26a13fdf034ca1fdd6abf60b46 mountpoint=/mnt/container namespace=k8s.io
DEBU[0000] container environment container-root=/var/lib/containerd image-root=/tmp/investigation manifest-file=
DEBU[0000] updated metadata file path=/tmp/investigation/var/lib/containerd/io.containerd.metadata.v1.bolt/meta.db
DEBU[0000] snapshot information for container 2df8c9bf881593eea104ed0cce6234582aef7f26a13fdf034ca1fdd6abf60b46 id=2df8c9bf881593eea104ed0cce6234582aef7f26a13fdf034ca1fdd6abf60b46 image="sha256:84e90d9540723be712a71a8ed21f082847b89f444d9c9c8a9d2696682d1f9848" snapshotkey=2df8c9bf881593eea104ed0cce6234582aef7f26a13fdf034ca1fdd6abf60b46 snapshotter=overlayfs
DEBU[0000] container root directory path=/tmp/investigation/var/lib/containerd
DEBU[0000] snapshotter root directory path=/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs
DEBU[0000] snapshotter database file path=/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/metadata.db
DEBU[0000] snapshotter root directory path=/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs
DEBU[0000] overlay directories lowerdir="/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/39/fs:/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/40/fs:/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/41/fs:/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/42/fs:/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/43/fs:/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/44/fs" upperdir=/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/45/fs workdir=/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/45/work
DEBU[0000] mount command options: [-t overlay overlay -o ro,lowerdir=/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/39/fs:/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/40/fs:/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/41/fs:/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/42/fs:/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/43/fs:/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/44/fs:/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/45/fs /mnt/container]
root@73e111c726f1:/go#
You see in the mount options that the fs directories are mounted as lowerdirs from 39-45, which is actually the wrong order.
The first commited layer of an image would overwrite everything that the other layers are doing to it.
This is proven by the follwing command:
root@73e111c726f1:/go# ls -la /mnt/container/home/SampleFiles/
total 8
drwxr-xr-x 2 root root 4096 Nov 15 23:58 .
drwxr-xr-x 1 root root 4096 Apr 15 2020 ..
root@73e111c726f1:/go#
The files shown above are not listed as they got deleted. They are present in layer 45 but not in 39, thus 39 will remove the files.
Solution
If you manually change the order in the mount command, everything is fine again:
# create mountpoint
mkdir /mnt/container
# mount dirs
mount -t overlay overlay -o ro,lowerdir=/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/45/fs:/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/44/fs:/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/43/fs:/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/42/fs:/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/41/fs:/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/40/fs:/tmp/investigation/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/39/fs /mnt/container
# List mount directory
ls -la /mnt/container/home/SampleFiles/
total 8336
drwxr-xr-x 1 root root 4096 Nov 17 17:59 .
drwxr-xr-x 1 root root 4096 Nov 15 23:58 ..
-rwxr-xr-x 1 root root 1089 Nov 15 23:50 DownloadFiles.sh
-rwxr-xr-x 1 root root 2155399 Nov 15 21:09 File01.2MB.tmp
-rwxr-xr-x 1 root root 2097180 Nov 15 21:09 File02.2MB.tmp
-rw-r--r-- 1 root root 2155399 Nov 17 17:59 File03.2MB.tmp
-rw-r--r-- 1 root root 2097180 Nov 17 17:59 File04.2MB.tmp
So long story short: You need to reverse the order of lowerdirs that you pass to mount. :)
Unfortunately i am not that skilled in Go and need the current time heavily to finish my thesis.. Otherwise i'd try to come up with a PR, so please have mercy.. :D
Kind regards,
pr3l14t0r
PS: If needed i can export the /var/lib/containerd directory of my exported image for you to test. But you can reproduce this with any other "once deployed" container that has an image using multiple layers.
