joincap
Merge multiple pcap files together, gracefully.
Installation
-
Download a precompiled binary from https://github.com/assafmo/joincap/releases
-
Or... Use
go get:go get -u github.com/assafmo/joincap
-
Or use Ubuntu PPA:
curl -SsL https://assafmo.github.io/ppa/ubuntu/KEY.gpg | sudo apt-key add - sudo curl -SsL -o /etc/apt/sources.list.d/assafmo.list https://assafmo.github.io/ppa/ubuntu/assafmo.list sudo apt update sudo apt install joincap
Basic Usage
Usage:
joincap [OPTIONS] InFiles...
Application Options:
-v, --verbose Explain when skipping packets or entire input files
-V, --version Print the version and exit
-w= Sets the output filename. If the name is '-', stdout will be used (default: -)
Help Options:
-h, --help Show this help message
Why?
I believe skipping corrupt packets is better than failing the entire merge job.
When using tcpslice or mergecap sometimes pcapfix is needed to fix bad input pcap files.
- One option is to try and run merge (
mergecap/tcpslice), if we get errors then runpcapfixon the bad pcaps and then run merge again.- Adds complexity (run -> check errors -> fix -> rerun)
- (If errors) Demands more resources (
pcapfixprocesses) - (If errors) Extends the total run time
- Another option is to run
pcapfixon the input pcap files and then merge.- Extends the total run time by a lot (read and write each pcap twice instead of once)
- Demands more storage (for the fixed pcaps)
- Demands more resources (
pcapfixprocesses)
- We can use
pcapfix"in memory" with process substitution:mergecap -w out.pcap <(pcapfix -o /dev/stdout 1.pcap) <(pcapfix -o /dev/stdout 2.pcap).- Adds complexity (build a complex command line)
- Demands more resources (
pcapfixprocesses) - Harder for us to use pathname expansion (e.g.
tcpslice -w out.pcap *.pcap) - We have to mind the command line character limit (in case of long pathnames)
- Doesn't work for
tcpslice(seeks the last packets to calculate time ranges - cannot do this with pipes)
Error handling: joincap vs mergecap vs tcpslice
Results
| Use case | joincap | mergecap v2.4.5 | tcpslice v1.2a3 |
|---|---|---|---|
| Corrupt input global header |
|
|
|
| Corrupt input packet header |
|
|
|
| Unexpectd EOF (last packet data is truncated) |
|
|
|
| Input pcap has no packets (global header is ok, no first packet header) |
|
|
|
| Input file size is smaller than 24 bytes (global header is truncated) |
|
|
|
| Input file size is between 24 and 40 bytes (global header is ok, first packet header is truncated) |
|
|
|
| Input file doesn't exists |
|
|
|
| Input file is a directory |
|
|
|
| Input file end is garbage |
|
|
|
| Input file is gzipped (.pcap.gz) |
|
|
|
Error outputs
| Use case | Error outputs |
|---|---|
| Corrupt input global header |
|
| Corrupt input packet header |
|
| Unexpectd EOF (last packet data is truncated) |
|
| Input pcap has no packets (global header is ok, no first packet header) |
|
| Input file size is smaller than 24 bytes (global header is truncated) |
|
| Input file size is between 24 and 40 bytes (global header is ok, first packet header is truncated) |
|
| Input file doesn't exists |
|
| Input file is a directory |
|
| Input file end is garbage |
|
| Input file is gzipped (.pcap.gz) |
|
How to reproduce
| Use case | How to reproduce |
|---|---|
| Corrupt input global header |
|
| Corrupt input packet header |
|
| Unexpectd EOF (last packet data is truncated) |
|
| Input pcap has no packets (global header is ok, no first packet header) |
|
| Input file size is smaller than 24 bytes (global header is truncated) |
|
| Input file size is between 24 and 40 bytes (global header is ok, first packet header is truncated) |
|
| Input file doesn't exists |
|
| Input file is a directory |
|
| Input file end is garbage |
|
| Input file is gzipped (.pcap.gz) |
|
Benchmarks
| Version | Speed | Time | |
|---|---|---|---|
| mergecap | 3.2.2 | 590MiB/s | 0m5.632s |
| tcpslice | 1.2a3 | 838MiB/s | 0m3.666s |
| joincap | 0.10.2 | 562MiB/s | 0m5.462s |
- Merging 3 files with total size of 2.99994GiB.
- Running on Linux 5.4.0-21-generic, with Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz (with SSE4.2), with 31765 MB of physical memory, with locale C, with zlib 1.2.11.
