A tool for capturing newly issued x.509 from Certificate Transparency logs & performing periodic revocation checking.

ct-logster

This repository contains the tools for collecting newly issued x509 certificates from Certificate Transparency logs, as well as performing OCSP & CRL revocation checks on the certficates.

logster

Logster is the tool that is used to monitor CT logs and fetch all newly issued certificates. The tool can listen to any amount of CT logs which reside in ctlogs.txt. Each log URL is separated by a new line, and should not start with https://.

Example ctlogs.txt file
yeti2022.ct.digicert.com/log
yeti2023.ct.digicert.com/log
ct.googleapis.com/logs/argon2021
ct.googleapis.com/logs/argon2022
ct.googleapis.com/logs/argon2023

From each newly issued certificate that is logged, the certificate PEM, as well as its chain is downloaded and stored in a MongoDB.

NOTE: A certificate can be logged to multiple CT logs. This tool does not take this into account and does NOT omit certificates that already have been stored in the database. As such, multiple entires of the same certificate will make it into the database.

revocado

Revocado is used to check the recovation statuses of certificates logged by logster. The primary way of checking revocation statuses is through OCSP requests. From these OCSP requests, we extract the revocation status, the revocation reason and the time of revocation. If OCSP isn't available, we perform a CRL check instead. The result from a CRL check is a binary yes/no, depending on if the certificate is in the CRL or not.

Error codes

A list of common error responses for OCSP requests are defined in DBhandler.go and have a corresponding error code:

unAuth  0
verErr  1
malFor  2 
badSig  3 
notOK   4
tOut    5
other   7

Setup

The tool is written in Go, and needs an installation of Go to compile. To compile the program, simply run go build.

The tool uses MongoDB for data storage and the configuration is stored in a .env file.

Example .env file
IP_ADDRESS="localhost"
PORT="27017"
DB="logs"
CERT_COLLECTION="chains"
USERNAME="username"
PASSWORD="passwd"

Performance

Depending on hardware used, the amount of parallell revocation checks that can be performed will differ. To control how many requests are ran concurrently, the semaphore sem in DBhandler.go is used. Feel free to play around with this to find a good value for your system.

cron

To set up periodic revocation checking, cron jobs can be used to run the tool every hour. To set up a crontab, run crontab -e and put in the following:

0 * * * * /path/to/ct-logster/logster/logster rev >> /path/to/ct-logster/logster/output.txt

This will run a revocation check every hour and append the output to output.txt.

Data structure

The data that is stored in the database is the struct CertInfo in logster.go. Feel free to remove/add fields to suit your needs.

type CertInfo struct {
	CertIndex    int            `bson:"certIndex"`
	SerialNumber string         `bson:"serialNumber"`
	Domain       []string       `bson:"domains"`
	OCSP         string         `bson:"OCSP,omitempty"`
	CRL          string         `bson:"CRL,omitempty"`
	CTlog        string         `bson:"ctLog"`
	Certificate  string         `bson:"cert,omitempty"`
	Chain        []string       `bson:"certChain,omitempty"`
	Changes      []StatusUpdate `bson:"Change"`
}

Running

To run the collection tool, simply run logster log. Do this for however long you wish to collect newly issued certificates. timeout can be used to set a time limit for the collection. For example, timeout 2d logster log will terminate the program after 2 days. Revocation checking can be performed in parallell with certificate collection, which is done easily by setting up cron jobs.

Notes

Were you to use our code, dataset, or parts of it in your work, we kindly ask that you reference the following paper in your publication:

  • Adam Halim, Max Danielsson, Martin Arlitt and Niklas Carlsson, "Temporal Analysis of X.509 Revocations and their Statuses", Proc. International Workshop on Traffic Measurements for Cybersecurity (WTMC), Genoa, Italy, June 2022
Similar Resources

EU Digital Covid Certificate utilities in Go [Create, Validate and Parse Green-Pass/EU-DCC]

go-dcc EU Digital Covid Certificate utilities in Go [Create, Validate and Parse Green-Pass/EU-DCC] Repo work in-progress CLI Usage: ######Create and S

Dec 23, 2021

Cert bound sts server - Certificate Bound Tokens using Security Token Exchange Server (STS)

Cert bound sts server - Certificate Bound Tokens using Security Token Exchange Server (STS)

Certificate Bound Tokens using Security Token Exchange Server (STS) Sample demonstration of Certificate Bound Tokens acquired from a Security Token Ex

Jan 2, 2022

Axiom Honeycomb Proxy ships logs to Axiom and Honeycomb simultaneously.

Axiom Honeycomb Proxy Table of Contents Introduction Usage Contributing License Introduction Axiom Honeycomb Proxy ships logs to Axiom and Honeycomb s

Dec 6, 2021

Watch for interesting patterns in Caddy logs and send a Telegram notification.

Watch for interesting patterns in Caddy logs and send a Telegram notification.

Jan 17, 2022

Telemetry interfaces for logs and metrics allowing complete decoupling of instrumentation implementations.

Telemetry This package provides a set of Telemetry interfaces allowing you to completely decouple your libraries and packages from Logging and Metrics

Apr 19, 2022

This is a tool that allows you to check minecraft names availability, this tool can do around 3000~ names a minute or more!

Checker This is a tool that allows you to check minecraft names availability, this tool can do around 3000~ names a minute or more! Tutorial To instal

Feb 13, 2022

Traefik config validator: a CLI tool to (syntactically) validate your Traefik configuration filesTraefik config validator: a CLI tool to (syntactically) validate your Traefik configuration files

Traefik config validator: a CLI tool to (syntactically) validate your Traefik configuration filesTraefik config validator: a CLI tool to (syntactically) validate your Traefik configuration files

Traefik Config Validator Note This is currently pre-release software. traefik-config-validator is a CLI tool to (syntactically) validate your Traefik

Dec 16, 2021

Server-tool - A simple tool to run and create Minecraft servers

Server Tool A simple tool to run and maintain different Minecraft servers. This

Jan 6, 2022
Related tags
Keeps track of Steam Deck Verifications. On first run, it reports all games with their respective Steam Deck Verification status. On subsequent runs, the tool will report newly tested and updated games.

Keeps track of Steam Deck Verifications. On first run, it reports all games with their respective Steam Deck Verification status. On subsequent runs, the tool will report newly tested and updated games.

Feb 2, 2022
Goket (Golang Keyboard Event Tree) is a proof-of-concept code for using keyboard events trees for performing operations.

Goket Goket (Golang Keyboard Event Tree) is a proof-of-concept code for using keyboard events trees for performing operations. Its main goal is to all

Jan 3, 2022
A tool for checking the accessibility of your data by IPFS peers

ipfs-check Check if you can find your content on IPFS A tool for checking the accessibility of your data by IPFS peers Documentation Build go build wi

Apr 18, 2022
PinGo is a standalone and feature-rich tool for common IP-based reachability checking tasks. Ping or Trace and Observe in real-time the statistics.

pingo As a network champion from designing and implementing to troubleshooting large scale networks - I know that is usually not easy for administrato

Feb 25, 2022
Go-aspell - GNU Aspell spell checking library bindings for golang

Aspell library bindings for Go GNU Aspell is a spell checking tool written in C/

Nov 29, 2021
Toy TLS certificate viewer

veilig Toy tls certificate viewer that I built because openssl s_client confuses me Source available at: https://github.com/noqqe/veilig/ Please repor

Aug 25, 2021
Go decoder for EU Digital COVID Certificate (EUDCC) QR code data

Go Corona QR Code Decoder This repository contains a decoder for EU Digital COVID Certificate (EUDCC) QR code data, written in Go. If you got vaccinat

Apr 11, 2022
DNS/DoT to DoH proxy with load-balancing, fail-over and SSL certificate management

dns-proxy Configuration Variable Example Description TLS_DOMAIN my.duckdns.org Domain name without wildcards. Used to create wildcard certificate and

Nov 23, 2021