Closes: https://github.com/palantir/log4j-sniffer/issues/72

This PR changes the way vulnerabilities are reported when not in --file-path-only mode.

Each finding is now reported individually, rather than reporting and aggregation of all findings with only the top-level file on disk.

For example, a vulnerable jar nested inside an archive will now be reported with the vulnerability findings, rather than reporting on the archive with an aggregation of all findings from within it. Multiple vulnerable jars found within an archive will be reported separately.

The path reported with a vulnerability finding is the full path to a finding with archive layers delimited by a "!". i.e. /path/to/archive!path/to/finding.jar shows that an archive at /path/to/archive contained a vulnerable jar at path/to/finding.jar within it.

To do this we now accept a HandleFindingFunc into the Log4jIdentifier, which is called for every finding encountered whilst crawling.

Reporter.Collect has also been renamed to Report to reflect better the behaviour that it has.

The standard go zip.Reader implementation creates a large slice of *zip.File when initialising the Reader. This is problematic for extremely large zip files as it can use a lot of memory.

This PR implements two zip walking functions that use a version of the standard go Reader to walk through all zip file contents without creating this large slice, using an iteration/stream approach instead.

To see the changes from the standard go zip package, compare the first commit against the following commits: https://github.com/palantir/log4j-sniffer/compare/9d735f4...gh/memory-considerate-zip-walker

Description of changes relative to the standard zip package codebase

Intent

The following modifications have been made to the original standard zip package to allow for iterating through zipped content in a memory considerate manner. The standard package has been optimised for reasons other than ours and the upstream implementation collects a slice of *File with a size equal to the number of files contained within a zip. This implementation works in a iterative/streaming manner where upon encountering a *File in the zip directory tree, the file is immediately passed to a function provided by the user for processing.

Package API

Two top-level functions have been introduced zip.WalkZipFile and WalkZipReaderAt that are akin to the standard package zip.OpenReader and zip.NewReader functions, respectively, but accepting another parameter, a WalkFn, and immediately walking through the contents of the zip file provided by the file or reader, passing each entry to the given WalkFn.

The WalkFn, when passed a *File should return a bool and an error. A false bool or a non-nil error will cause the walk to stop and return the error to the user.

Changes made relative to the codebase in the standard zip package

• All Write-related code has been removed where possible. We are only interested in reading.
• All ways ways to create a Reader/ReadCloser have been removed from the package API, allowing only a single entrypoint to the Walk functions that we have implemented.
• After making the above changes, all unused code has been removed apart from some constants that were part of constant groups.
• The Reader.init method has been renamed to Reader.walk to reperesent the functionality that it now holds. More details on the cahnges for this can be found below.
• The Reader.Files field, a []*File has been removed to make it impossible to incur the memory penalty of accumalating a slice of *File details.
• All tests that are appropriate to keep have been migrated to represent equivalents of their original but using the new Walk methods. An extra test file reader_local_test.go has been introduced, containing tests specific for our implementation.
• Some tests that use standard internal package have been removed as it is impossible for us to reference them Examples have been removed.
• Some general changes have been made to make the codebase meet some of our linter requirements. i.e. Matching names of method receivers in method definitions, some extra error checking and removal of unnecessary conversions between value types.

Changes to Reader.init/Reader.walk

• Reader.init is where the standard package reads through the directory tree of the zip file to accumulate a slice of *File for each file in the zip. This method has been changed to accept a WalkFn which each *File is passed to instead of appending it to a slice. The WalkFn is passed into the two top-level Walk functions by the caller.
• All logic related to Reader.Files has been removed.
• The given WalkFn, when passed a *File should return a bool and an error. A false bool or a non-nil error will cause the walk to stop and return the error to the user.
• If the WalkFn never returns a false or non-nil error, the expected number of directory records is compared against the number of files iterated through, in a similar manner that the length of Reader.Files was previously checked. If the numbers do not match, an error is returned from the top-level Walk functions.
• File.readDataDescriptor is no longer called on each file as it is iterated, this change has been made for performance reasons as it is an expensive call and the information populated by File.readDataDescriptor is not required unless the File is being open. This is now called within the File.Open method so that it is only called if the file is going to be opened. A user can now determine whether the file should be opened from the filename or size, for example, and then only have f.readDataDescriptor called for those that are opened.

Changes to File

• File.readDataDescriptor now accepts a bodyOffset rather than calling findBodyOffset to determine it. This change was made because in File.Open, we now find the body offset when Open is called and can inject it into readDataDescriptor to avoid findBodyOffset being called multiple times per Open.

Adds a --archive-open-mode flag with supported values of "standard" and "directio", defaulting to "standard".

Standard open mode will open files and read their content in the same way as before this PR, where the filesystem cache may be used. This has caused issues on some hosts, where the use of log4j-sniffer has resulted in many filesystem cache evictions during use.

Direct i/o mode will use the O_DIRECT flag when opening archives on disk on linux systems, using similar primitives on other operating systems that allow for reading of the files to skip the cache.

Closes: https://github.com/palantir/log4j-sniffer/issues/77

The description of V0.5.0 says: "log4j-sniffer crawls for all instances of log4j that are earlier than version 2.16 on disk"

But the recommended version of Log4J is 2.17 so 2.16 should also be considered unsafe to some degree as it is vulnerable to a DoS attack (CVE-2021-45105).

Thanks for a great tool!!!

This adds a command that implements a basic approach bytecode instruction identification:

• Parse the classfile format
• Loop over all methods as they appear in the classfile
• For the code in each method, take the opcodes only (i.e. no operands/variable bytes) and run them through md5

Running on log4j-core binaries this produces distinct hashes for 2.16.0, 2.15.0 and 2.12.2. 2.12.0 through 2.14.1 result in the same hash, 2.11.2 is different. Earlier versions have not been tested at this time.

Currently, the program identifies vulnerabilities by matching based on file names and classes. However, this could be improved by analyzing the content to detect cases in which JARs are shaded and/or slightly modified by frameworks such as ProGuard.

Closes https://github.com/palantir/log4j-sniffer/issues/40

Adds a feature that can be enabled with by setting --nested-archive-disk-swap-max-size to a positive non-zero value.

When a nested zip file is encountered that is above the --nested-archive-max-size, space will be used on disk to write out the archive temporarily to be able to inspected without incurring a large memory penalty. Deeply nested archives can be temporarily written out to disk, even if the parent archives are written to disk, providing that the total space taken up is less than or equal to --nested-archive-disk-swap-max-size.

In https://github.com/palantir/log4j-sniffer/pull/93 there was a small bug introduced where the directory rather than whole filepath would be considered for the --filepath-owner option.

We can reuse buffers in the Reader struct to reduce the number of allocations when walking a zip. The pointer passed to the handler will now point to the same memory address but we have already stated that the file should only be used during the duration of the WalkFn callback so this is fine.

The largest improvements are for zips containing many files but we still get a 15% speedup from zips containing as little as 10 files. These optimisations only improve the overhead of scanning a zip file, not processing contents etc.

$ benchstat /tmp/before /tmp/after
name                       old time/op    new time/op    delta
WalkZipReaderAt/10-16        6.03µs ± 1%    5.09µs ± 3%  -15.66%  (p=0.004 n=5+6)
WalkZipReaderAt/100-16       24.2µs ± 1%    14.8µs ± 2%  -39.03%  (p=0.002 n=6+6)
WalkZipReaderAt/1000-16       215µs ± 2%     109µs ± 0%  -49.25%  (p=0.004 n=6+5)
WalkZipReaderAt/10000-16     2.31ms ± 1%    1.11ms ± 1%  -51.76%  (p=0.004 n=6+5)
WalkZipReaderAt/100000-16    21.6ms ± 2%    11.7ms ± 2%  -46.06%  (p=0.002 n=6+6)

name                       old alloc/op   new alloc/op   delta
WalkZipReaderAt/10-16        8.08kB ± 0%    5.70kB ± 0%  -29.50%  (p=0.002 n=6+6)
WalkZipReaderAt/100-16       30.0kB ± 0%     5.9kB ± 0%  -80.45%  (p=0.002 n=6+6)
WalkZipReaderAt/1000-16       252kB ± 0%       9kB ± 0%  -96.51%  (p=0.002 n=6+6)
WalkZipReaderAt/10000-16     2.48MB ± 0%    0.04MB ± 0%  -98.20%  (p=0.000 n=5+6)
WalkZipReaderAt/100000-16    25.0MB ± 0%     0.5MB ± 0%  -97.90%  (p=0.002 n=6+6)

name                       old allocs/op  new allocs/op  delta
WalkZipReaderAt/10-16          39.0 ± 0%       9.0 ± 0%  -76.92%  (p=0.002 n=6+6)
WalkZipReaderAt/100-16          399 ± 0%       100 ± 0%  -74.94%  (p=0.002 n=6+6)
WalkZipReaderAt/1000-16       4.00k ± 0%     1.00k ± 0%  -74.93%  (p=0.002 n=6+6)
WalkZipReaderAt/10000-16      40.0k ± 0%     10.0k ± 0%  -74.99%  (p=0.002 n=6+6)
WalkZipReaderAt/100000-16      400k ± 0%      100k ± 0%  -75.00%  (p=0.002 n=6+6)

Before: in the case of a jar with mostly non-obfuscated classes with a small number of obfuscated mixed in we'd not examine any class files a the average package and class name length would be skewed by the non-obfuscated classes After: we consider each class individually as to whether it is obfuscated based on the packages and class name of that class only.

This has the huge benefit of eliminating the initial iteration through all jar filenames, which is expensive in terms of IO and currently memory.

Top-level Crawl signature change Crawl now accepts a type ProcessFunc func(ctx context.Context, path string, result Finding, version Versions) handler responsible for reporting on individual findings during the crawl.

The ProcessFunc used in the production code is the same reporter that has been previously used, the driver of this change was to produce a friendly framework for testing examples, allowing us to more easily add specific test cases and test against an actual data structure rather than string matching against command output.

crawl.Stats Also in this change, the top-level Crawl function now passes the crawl.Stats from the generic filesystem crawler to the crawl command, where the summary rendering logic now lives.

Simple summary rendering test Adds a simple integrations test for the summary printing so that we can ensure we don't break the output in case there are system log queries or workflows based on that specific line.

I installed "go" and "log4j-sniffer" successfully. Tried to run "log4j-sniffer" in UbuntuLinux from my Sysadmin folder and got a message "command not found". Please advise.

It would help if we had a way to suppress these matches (produced by the crawl command) when the application owner has upgraded to log4j-core-2.17.1.jar but renamed the file.

In this example, I scanned examples/good_version/log4j-core-2.17.1.jar but removed "core-" from the name.

[MATCH] invalid version - unknown CVE status detected in file log4j-2.17.1.jar. log4j versions: unknown. Reasons: JndiLookup class and package name matched

The tool doesn't ignore subdirectories on Windows hosts, using the --ignore-dir argument as currently documented:

      --ignore-dir strings                 Specify directory pattern to ignore. Use multiple times to supply multiple patterns.
                                           Patterns should be relative to the provided root.
                                           e.g. ignore "^/proc" to ignore "/proc" when using a crawl root of "/"

Test setup

two subdirectories containing vulnerable jars:

PS > tree /f
Folder PATH listing
C:.
│   log4j-sniffer-0.8.0-windows-amd64.exe
│
├───test1
│       apache-log4j-2.14.0-bin.zip
│
└───test2
    └───subdir
            apache-log4j-2.14.0-bin.zip

Supplied syntax doesn't ignore test1 (with either \ or /)

PS > .\log4j-sniffer-0.8.0-windows-amd64.exe crawl .\ --ignore-dir '^/test1'
CVE-2021-45046 and CVE-2021-45105 detected in file test1\apache-log4j-2.14.0-bin.zip. log4j versions: 2.14.0. Reasons: jar name inside archive matched
CVE-2021-45046 and CVE-2021-45105 detected in file test2\subdir\apache-log4j-2.14.0-bin.zip. log4j versions: 2.14.0. Reasons: jar name inside archive matched
Files affected by CVE-2021-45046 or CVE-2021-45105 detected: 2 file(s) impacted by CVE-2021-45046 or CVE-2021-45105
3 total files scanned, skipped 0 paths due to permission denied errors, encountered 0 errors processing paths
PS > .\log4j-sniffer-0.8.0-windows-amd64.exe crawl .\ --ignore-dir '^\test1'
CVE-2021-45046 and CVE-2021-45105 detected in file test1\apache-log4j-2.14.0-bin.zip. log4j versions: 2.14.0. Reasons: jar name inside archive matched
CVE-2021-45046 and CVE-2021-45105 detected in file test2\subdir\apache-log4j-2.14.0-bin.zip. log4j versions: 2.14.0. Reasons: jar name inside archive matched
Files affected by CVE-2021-45046 or CVE-2021-45105 detected: 2 file(s) impacted by CVE-2021-45046 or CVE-2021-45105
3 total files scanned, skipped 0 paths due to permission denied errors, encountered 0 errors processing paths

A single-level subdirectory can be ignored by just specifying its name with no ^ / or \:

PS> .\log4j-sniffer-0.8.0-windows-amd64.exe crawl .\ --ignore-dir 'test1'
CVE-2021-45046 and CVE-2021-45105 detected in file test2\subdir\apache-log4j-2.14.0-bin.zip. log4j versions: 2.14.0. Reasons: jar name inside archive matched
Files affected by CVE-2021-45046 or CVE-2021-45105 detected: 1 file(s) impacted by CVE-2021-45046 or CVE-2021-45105
2 total files scanned, skipped 0 paths due to permission denied errors, encountered 0 errors processing paths

This doesn't work for nested subdirectories:

PS > .\log4j-sniffer-0.8.0-windows-amd64.exe crawl .\ --ignore-dir 'test2\subdir'
CVE-2021-45046 and CVE-2021-45105 detected in file test1\apache-log4j-2.14.0-bin.zip. log4j versions: 2.14.0. Reasons: jar name inside archive matched
CVE-2021-45046 and CVE-2021-45105 detected in file test2\subdir\apache-log4j-2.14.0-bin.zip. log4j versions: 2.14.0. Reasons: jar name inside archive matched
Files affected by CVE-2021-45046 or CVE-2021-45105 detected: 2 file(s) impacted by CVE-2021-45046 or CVE-2021-45105
3 total files scanned, skipped 0 paths due to permission denied errors, encountered 0 errors processing paths
PS > .\log4j-sniffer-0.8.0-windows-amd64.exe crawl .\ --ignore-dir 'test2/subdir'
CVE-2021-45046 and CVE-2021-45105 detected in file test1\apache-log4j-2.14.0-bin.zip. log4j versions: 2.14.0. Reasons: jar name inside archive matched
CVE-2021-45046 and CVE-2021-45105 detected in file test2\subdir\apache-log4j-2.14.0-bin.zip. log4j versions: 2.14.0. Reasons: jar name inside archive matched
Files affected by CVE-2021-45046 or CVE-2021-45105 detected: 2 file(s) impacted by CVE-2021-45046 or CVE-2021-45105
3 total files scanned, skipped 0 paths due to permission denied errors, encountered 0 errors processing paths

Implements a docker command that scans all locally stored images from the docker daemon. The docker functionality is purely focused on exporting the image tarballs to disk and scanning them using the standard filesystem crawler so there should be no nuances about what can be detected in docker vs filesystem.

Scanning is initiated by the ScanImages function which takes in a scan configuration along with a docker client (if docker is not running the client creation fails earlier on), the first step is to get a list of images from the daemon using the client, the returned images are iterated over and each one processed by the scanImage function where we use go-containerregistry/crane to flatten the images to a single layer and then export the image as a tarball, from there we extract the image tarball to disk and use the filesystem crawler from the extracted image tarball directory to perform a regular scan, on completion all created files are removed.

Sample output

CVE-2021-45046 and CVE-2021-45105 detected in image df67a306e829 [log4j/log4j-vuln-jars:latest] in file opt/shadow-all.jar. log4j versions: 2.12.0 - 2.14.1. Reasons: class name matched, byte code instruction MD5 matched
CVE-2021-45046 and CVE-2021-45105 detected in image df67a306e829 [log4j/log4j-vuln-jars:latest] in file opt/wrapped_log4j.tar. log4j versions: 2.14.1. Reasons: jar name inside archive matched
CVE-2021-45046 and CVE-2021-45105 detected in image df67a306e829 [log4j/log4j-vuln-jars:latest] in file opt/wrapped_log4j.tar.bz2. log4j versions: 2.14.1. Reasons: jar name inside archive matched
CVE-2021-45046 and CVE-2021-45105 detected in image df67a306e829 [log4j/log4j-vuln-jars:latest] in file opt/wrapped_log4j.tar.gz. log4j versions: 2.14.1. Reasons: jar name inside archive matched
CVE-2021-45046 and CVE-2021-45105 detected in image df67a306e829 [log4j/log4j-vuln-jars:latest] in file opt/wrapped_log4j.zip. log4j versions: 2.14.1. Reasons: jar name inside archive matched
CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/cve-2021-45105-versions/log4j-core-2.12.2.jar. log4j versions: 2.12.2. Reasons: jar name matched, class and package name matched, class file MD5 matched
CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/cve-2021-45105-versions/log4j-core-2.16.0.jar. log4j versions: 2.16.0. Reasons: jar name matched, class and package name matched, class file MD5 matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/fat_jar/fat_jar.jar. log4j versions: 2.14.0 - 2.14.1. Reasons: class and package name matched, class file MD5 matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/inside_a_dist/wrapped_log4j.tar. log4j versions: 2.14.1. Reasons: jar name inside archive matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/inside_a_dist/wrapped_log4j.tar.bz2. log4j versions: 2.14.1. Reasons: jar name inside archive matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/inside_a_dist/wrapped_log4j.tar.gz. log4j versions: 2.14.1. Reasons: jar name inside archive matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/inside_a_dist/wrapped_log4j.zip. log4j versions: 2.14.1. Reasons: jar name inside archive matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/inside_a_par/wrapped_in_a_par.par. log4j versions: 2.14.1. Reasons: jar name inside archive matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/light_shading/shadow-all.jar. log4j versions: 2.12.0 - 2.14.1. Reasons: class name matched, byte code instruction MD5 matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/multiple_bad_versions/log4j-core-2.10.0.jar. log4j versions: 2.10.0, 2.9.0-2.11.2. Reasons: jar name matched, class and package name matched, class file MD5 matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/multiple_bad_versions/log4j-core-2.11.0.jar. log4j versions: 2.11.0, 2.9.0-2.11.2. Reasons: jar name matched, class and package name matched, class file MD5 matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/multiple_bad_versions/log4j-core-2.11.1.jar. log4j versions: 2.11.1, 2.9.0-2.11.2. Reasons: jar name matched, class and package name matched, class file MD5 matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/multiple_bad_versions/log4j-core-2.11.2.jar. log4j versions: 2.11.2, 2.9.0-2.11.2. Reasons: jar name matched, class and package name matched, class file MD5 matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/multiple_bad_versions/log4j-core-2.12.0.jar. log4j versions: 2.12.0. Reasons: jar name matched, class and package name matched, class file MD5 matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/multiple_bad_versions/log4j-core-2.12.1.jar. log4j versions: 2.12.0, 2.12.1. Reasons: jar name matched, class and package name matched, class file MD5 matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/multiple_bad_versions/log4j-core-2.13.0.jar. log4j versions: 2.13.0, 2.13.0-2.13.3. Reasons: jar name matched, class and package name matched, class file MD5 matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/multiple_bad_versions/log4j-core-2.13.1.jar. log4j versions: 2.13.0-2.13.3, 2.13.1. Reasons: jar name matched, class and package name matched, class file MD5 matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/multiple_bad_versions/log4j-core-2.13.2.jar. log4j versions: 2.13.0-2.13.3, 2.13.2. Reasons: jar name matched, class and package name matched, class file MD5 matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/multiple_bad_versions/log4j-core-2.13.3.jar. log4j versions: 2.13.0-2.13.3, 2.13.3. Reasons: jar name matched, class and package name matched, class file MD5 matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/multiple_bad_versions/log4j-core-2.14.0.jar. log4j versions: 2.14.0, 2.14.0 - 2.14.1. Reasons: jar name matched, class and package name matched, class file MD5 matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/multiple_bad_versions/log4j-core-2.14.1.jar. log4j versions: 2.14.0 - 2.14.1, 2.14.1. Reasons: jar name matched, class and package name matched, class file MD5 matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/multiple_bad_versions/log4j-core-2.15.0.jar. log4j versions: 2.15.0. Reasons: jar name matched, class and package name matched, class file MD5 matched
CVE-2021-45046 and CVE-2021-45105 detected in image f98b754f47fa [log4j/log4j-multiple:latest] in file opt/single_bad_version/log4j-core-2.14.1.jar. log4j versions: 2.14.0 - 2.14.1, 2.14.1. Reasons: jar name matched, class and package name matched, class file MD5 matched
Files affected by CVE-2021-45046 or CVE-2021-45105 detected: 28 file(s) impacted by CVE-2021-45046 or CVE-2021-45105
45 total files scanned, skipped 0 paths due to permission denied errors, encountered 0 errors processing paths