Automatic HTTPS for any Go program: fully-managed TLS certificate issuance and renewal

CertMagic

Easy and Powerful TLS Automation

The same library used by the Caddy Web Server

Caddy's automagic TLS features—now for your own Go programs—in one powerful and easy-to-use library!

CertMagic is the most mature, robust, and capable ACME client integration for Go... and perhaps ever.

With CertMagic, you can add one line to your Go application to serve securely over TLS, without ever having to touch certificates.

Instead of:

// plaintext HTTP, gross 🤢
http.ListenAndServe(":80", mux)

Use CertMagic:

// encrypted HTTPS with HTTP->HTTPS redirects - yay! 🔒😍
certmagic.HTTPS([]string{"example.com"}, mux)

That line of code will serve your HTTP router mux over HTTPS, complete with HTTP->HTTPS redirects. It obtains and renews the TLS certificates. It staples OCSP responses for greater privacy and security. As long as your domain name points to your server, CertMagic will keep its connections secure.

Compared to other ACME client libraries for Go, only CertMagic supports the full suite of ACME features, and no other library matches CertMagic's maturity and reliability.

CertMagic - Automatic HTTPS using Let's Encrypt

Sponsored by Relica - Cross-platform local and cloud file backup:

Relica - Cross-platform file backup to the cloud, local disks, or other computers

Menu

Features

  • Fully automated certificate management including issuance and renewal
  • One-liner, fully managed HTTPS servers
  • Full control over almost every aspect of the system
  • HTTP->HTTPS redirects
  • Solves all 3 ACME challenges: HTTP, TLS-ALPN, and DNS
  • Most robust error handling of any ACME client
    • Challenges are randomized to avoid accidental dependence
    • Challenges are rotated to overcome certain network blockages
    • Robust retries for up to 30 days
    • Exponential backoff with carefully-tuned intervals
    • Retries with optional test/staging CA endpoint instead of production, to avoid rate limits
  • Written in Go, a language with memory-safety guarantees
  • Powered by ACMEz, the premier ACME client library for Go
  • All libdns DNS providers work out-of-the-box
  • Pluggable storage implementations (default: file system)
  • Wildcard certificates
  • Automatic OCSP stapling (done right) keeps your sites online!
  • Distributed solving of all challenges (works behind load balancers)
    • Highly efficient, coordinated management in a fleet
    • Active locking
    • Smart queueing
  • Supports "on-demand" issuance of certificates (during TLS handshakes!)
    • Caddy / CertMagic pioneered this technology
    • Custom decision functions to regulate and throttle on-demand behavior
  • Optional event hooks for observation
  • Works with any certificate authority (CA) compliant with the ACME specification
  • Certificate revocation (please, only if private key is compromised)
  • Must-Staple (optional; not default)
  • Cross-platform support! Mac, Windows, Linux, BSD, Android...
  • Scales to hundreds of thousands of names/certificates per instance
  • Use in conjunction with your own certificates

Requirements

  1. Public DNS name(s) you control
  2. Server reachable from public Internet
    • Or use the DNS challenge to waive this requirement
  3. Control over port 80 (HTTP) and/or 443 (HTTPS)
    • Or they can be forwarded to other ports you control
    • Or use the DNS challenge to waive this requirement
    • (This is a requirement of the ACME protocol, not a library limitation)
  4. Persistent storage
    • Typically the local file system (default)
    • Other integrations available/possible

Before using this library, your domain names MUST be pointed (A/AAAA records) at your server (unless you use the DNS challenge)!

Installation

$ go get github.com/caddyserver/certmagic

Usage

Package Overview

Certificate authority

This library uses Let's Encrypt by default, but you can use any certificate authority that conforms to the ACME specification. Known/common CAs are provided as consts in the package, for example LetsEncryptStagingCA and LetsEncryptProductionCA.

The Config type

The certmagic.Config struct is how you can wield the power of this fully armed and operational battle station. However, an empty/uninitialized Config is not a valid one! In time, you will learn to use the force of certmagic.NewDefault() as I have.

Defaults

The default Config value is called certmagic.Default. Change its fields to suit your needs, then call certmagic.NewDefault() when you need a valid Config value. In other words, certmagic.Default is a template and is not valid for use directly.

You can set the default values easily, for example: certmagic.Default.Issuer = ....

Similarly, to configure ACME-specific defaults, use certmagic.DefaultACME.

The high-level functions in this package (HTTPS(), Listen(), ManageSync(), and ManageAsync()) use the default config exclusively. This is how most of you will interact with the package. This is suitable when all your certificates are managed the same way. However, if you need to manage certificates differently depending on their name, you will need to make your own cache and configs (keep reading).

Providing an email address

Although not strictly required, this is highly recommended best practice. It allows you to receive expiration emails if your certificates are expiring for some reason, and also allows the CA's engineers to potentially get in touch with you if something is wrong. I recommend setting certmagic.DefaultACME.Email or always setting the Email field of a new Config struct.

Rate limiting

To avoid firehosing the CA's servers, CertMagic has built-in rate limiting. Currently, its default limit is up to 10 transactions (obtain or renew) every 1 minute (sliding window). This can be changed by setting the RateLimitEvents and RateLimitEventsWindow variables, if desired.

The CA may still enforce their own rate limits, and there's nothing (well, nothing ethical) CertMagic can do to bypass them for you.

Additionally, CertMagic will retry failed validations with exponential backoff for up to 30 days, with a reasonable maximum interval between attempts (an "attempt" means trying each enabled challenge type once).

Development and Testing

Note that Let's Encrypt imposes strict rate limits at its production endpoint, so using it while developing your application may lock you out for a few days if you aren't careful!

While developing your application and testing it, use their staging endpoint which has much higher rate limits. Even then, don't hammer it: but it's much safer for when you're testing. When deploying, though, use their production CA because their staging CA doesn't issue trusted certificates.

To use staging, set certmagic.DefaultACME.CA = certmagic.LetsEncryptStagingCA or set CA of every ACMEManager struct.

Examples

There are many ways to use this library. We'll start with the highest-level (simplest) and work down (more control).

All these high-level examples use certmagic.Default and certmagic.DefaultACME for the config and the default cache and storage for serving up certificates.

First, we'll follow best practices and do the following:

// read and agree to your CA's legal documents
certmagic.DefaultACME.Agreed = true

// provide an email address
certmagic.DefaultACME.Email = "[email protected]"

// use the staging endpoint while we're developing
certmagic.DefaultACME.CA = certmagic.LetsEncryptStagingCA

For fully-functional program examples, check out this Twitter thread (or read it unrolled into a single post). (Note that the package API has changed slightly since these posts.)

Serving HTTP handlers with HTTPS

err := certmagic.HTTPS([]string{"example.com", "www.example.com"}, mux)
if err != nil {
	return err
}

This starts HTTP and HTTPS listeners and redirects HTTP to HTTPS!

Starting a TLS listener

ln, err := certmagic.Listen([]string{"example.com"})
if err != nil {
	return err
}

Getting a tls.Config

tlsConfig, err := certmagic.TLS([]string{"example.com"})
if err != nil {
	return err
}

Advanced use

For more control (particularly, if you need a different way of managing each certificate), you'll make and use a Cache and a Config like so:

cache := certmagic.NewCache(certmagic.CacheOptions{
	GetConfigForCert: func(cert certmagic.Certificate) (*certmagic.Config, error) {
		// do whatever you need to do to get the right
		// configuration for this certificate; keep in
		// mind that this config value is used as a
		// template, and will be completed with any
		// defaults that are set in the Default config
		return &certmagic.Config{
			// ...
		}, nil
	},
	...
})

magic := certmagic.New(cache, certmagic.Config{
	// any customizations you need go here
})

myACME := certmagic.NewACMEManager(magic, ACMEManager{
	CA:     certmagic.LetsEncryptStagingCA,
	Email:  "[email protected]",
	Agreed: true,
	// plus any other customizations you need
})

magic.Issuer = myACME

// this obtains certificates or renews them if necessary
err := magic.ManageSync([]string{"example.com", "sub.example.com"})
if err != nil {
	return err
}

// to use its certificates and solve the TLS-ALPN challenge,
// you can get a TLS config to use in a TLS listener!
tlsConfig := magic.TLSConfig()

//// OR ////

// if you already have a TLS config you don't want to replace,
// we can simply set its GetCertificate field and append the
// TLS-ALPN challenge protocol to the NextProtos
myTLSConfig.GetCertificate = magic.GetCertificate
myTLSConfig.NextProtos = append(myTLSConfig.NextProtos, tlsalpn01.ACMETLS1Protocol}

// the HTTP challenge has to be handled by your HTTP server;
// if you don't have one, you should have disabled it earlier
// when you made the certmagic.Config
httpMux = myACME.HTTPChallengeHandler(httpMux)

Great! This example grants you much more flexibility for advanced programs. However, the vast majority of you will only use the high-level functions described earlier, especially since you can still customize them by setting the package-level Default config.

Wildcard certificates

At time of writing (December 2018), Let's Encrypt only issues wildcard certificates with the DNS challenge. You can easily enable the DNS challenge with CertMagic for numerous providers (see the relevant section in the docs).

Behind a load balancer (or in a cluster)

CertMagic runs effectively behind load balancers and/or in cluster/fleet environments. In other words, you can have 10 or 1,000 servers all serving the same domain names, all sharing certificates and OCSP staples.

To do so, simply ensure that each instance is using the same Storage. That is the sole criteria for determining whether an instance is part of a cluster.

The default Storage is implemented using the file system, so mounting the same shared folder is sufficient (see Storage for more on that)! If you need an alternate Storage implementation, feel free to use one, provided that all the instances use the same one. :)

See Storage and the associated pkg.go.dev for more information!

The ACME Challenges

This section describes how to solve the ACME challenges. Challenges are how you demonstrate to the certificate authority some control over your domain name, thus authorizing them to grant you a certificate for that name. The great innovation of ACME is that verification by CAs can now be automated, rather than having to click links in emails (who ever thought that was a good idea??).

If you're using the high-level convenience functions like HTTPS(), Listen(), or TLS(), the HTTP and/or TLS-ALPN challenges are solved for you because they also start listeners. However, if you're making a Config and you start your own server manually, you'll need to be sure the ACME challenges can be solved so certificates can be renewed.

The HTTP and TLS-ALPN challenges are the defaults because they don't require configuration from you, but they require that your server is accessible from external IPs on low ports. If that is not possible in your situation, you can enable the DNS challenge, which will disable the HTTP and TLS-ALPN challenges and use the DNS challenge exclusively.

Technically, only one challenge needs to be enabled for things to work, but using multiple is good for reliability in case a challenge is discontinued by the CA. This happened to the TLS-SNI challenge in early 2018—many popular ACME clients such as Traefik and Autocert broke, resulting in downtime for some sites, until new releases were made and patches deployed, because they used only one challenge; Caddy, however—this library's forerunner—was unaffected because it also used the HTTP challenge. If multiple challenges are enabled, they are chosen randomly to help prevent false reliance on a single challenge type. And if one fails, any remaining enabled challenges are tried before giving up.

HTTP Challenge

Per the ACME spec, the HTTP challenge requires port 80, or at least packet forwarding from port 80. It works by serving a specific HTTP response that only the genuine server would have to a normal HTTP request at a special endpoint.

If you are running an HTTP server, solving this challenge is very easy: just wrap your handler in HTTPChallengeHandler or call SolveHTTPChallenge() inside your own ServeHTTP() method.

For example, if you're using the standard library:

mux := http.NewServeMux()
mux.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
	fmt.Fprintf(w, "Lookit my cool website over HTTPS!")
})

http.ListenAndServe(":80", myACME.HTTPChallengeHandler(mux))

If wrapping your handler is not a good solution, try this inside your ServeHTTP() instead:

magic := certmagic.NewDefault()
myACME := certmagic.NewACMEManager(magic, certmagic.DefaultACME)

func ServeHTTP(w http.ResponseWriter, req *http.Request) {
	if myACME.HandleHTTPChallenge(w, r) {
		return // challenge handled; nothing else to do
	}
	...
}

If you are not running an HTTP server, you should disable the HTTP challenge or run an HTTP server whose sole job it is to solve the HTTP challenge.

TLS-ALPN Challenge

Per the ACME spec, the TLS-ALPN challenge requires port 443, or at least packet forwarding from port 443. It works by providing a special certificate using a standard TLS extension, Application Layer Protocol Negotiation (ALPN), having a special value. This is the most convenient challenge type because it usually requires no extra configuration and uses the standard TLS port which is where the certificates are used, also.

This challenge is easy to solve: just use the provided tls.Config when you make your TLS listener:

// use this to configure a TLS listener
tlsConfig := magic.TLSConfig()

Or make two simple changes to an existing tls.Config:

myTLSConfig.GetCertificate = magic.GetCertificate
myTLSConfig.NextProtos = append(myTLSConfig.NextProtos, tlsalpn01.ACMETLS1Protocol}

Then just make sure your TLS listener is listening on port 443:

ln, err := tls.Listen("tcp", ":443", myTLSConfig)

DNS Challenge

The DNS challenge is perhaps the most useful challenge because it allows you to obtain certificates without your server needing to be publicly accessible on the Internet, and it's the only challenge by which Let's Encrypt will issue wildcard certificates.

This challenge works by setting a special record in the domain's zone. To do this automatically, your DNS provider needs to offer an API by which changes can be made to domain names, and the changes need to take effect immediately for best results. CertMagic supports all DNS providers with libdns implementations! It always cleans up the temporary record after the challenge completes.

To enable it, just set the DNS01Solver field on a certmagic.ACMEManager struct, or set the default certmagic.ACMEManager.DNS01Solver variable. For example, if my domains' DNS was served by Cloudflare:

import "github.com/libdns/cloudflare"

certmagic.DefaultACME.DNS01Solver = &certmagic.DNS01Solver{
	DNSProvider: &cloudflare.Provider{
		APIToken: "topsecret",
	},
}

Now the DNS challenge will be used by default, and I can obtain certificates for wildcard domains, too. Enabling the DNS challenge disables the other challenges for that certmagic.ACMEManager instance.

On-Demand TLS

Normally, certificates are obtained and renewed before a listener starts serving, and then those certificates are maintained throughout the lifetime of the program. In other words, the certificate names are static. But sometimes you don't know all the names ahead of time, or you don't want to manage all the certificates up front. This is where On-Demand TLS shines.

Originally invented for use in Caddy (which was the first program to use such technology), On-Demand TLS makes it possible and easy to serve certificates for arbitrary or specific names during the lifetime of the server. When a TLS handshake is received, CertMagic will read the Server Name Indication (SNI) value and either load and present that certificate in the ServerHello, or if one does not exist, it will obtain it from a CA right then-and-there.

Of course, this has some obvious security implications. You don't want to DoS a CA or allow arbitrary clients to fill your storage with spammy TLS handshakes. That's why, when you enable On-Demand issuance, you should set limits or policy to allow getting certificates. CertMagic has an implicit whitelist built-in which is sufficient for nearly everyone, but also has a more advanced way to control on-demand issuance.

The simplest way to enable on-demand issuance is to set the OnDemand field of a Config (or the default package-level value):

certmagic.Default.OnDemand = new(certmagic.OnDemandConfig)

By setting this to a non-nil value, on-demand TLS is enabled for that config. For convenient security, CertMagic's high-level abstraction functions such as HTTPS(), TLS(), ManageSync(), ManageAsync(), and Listen() (which all accept a list of domain names) will whitelist those names automatically so only certificates for those names can be obtained when using the Default config. Usually this is sufficient for most users.

However, if you require advanced control over which domains can be issued certificates on-demand (for example, if you do not know which domain names you are managing, or just need to defer their operations until later), you should implement your own DecisionFunc:

// if the decision function returns an error, a certificate
// may not be obtained for that name at that time
certmagic.Default.OnDemand = &certmagic.OnDemandConfig{
	DecisionFunc: func(name string) error {
		if name != "example.com" {
			return fmt.Errorf("not allowed")
		}
		return nil
	},
}

The pkg.go.dev describes how to use this in full detail, so please check it out!

Storage

CertMagic relies on storage to store certificates and other TLS assets (OCSP staple cache, coordinating locks, etc). Persistent storage is a requirement when using CertMagic: ephemeral storage will likely lead to rate limiting on the CA-side as CertMagic will always have to get new certificates.

By default, CertMagic stores assets on the local file system in $HOME/.local/share/certmagic (and honors $XDG_DATA_HOME if set). CertMagic will create the directory if it does not exist. If writes are denied, things will not be happy, so make sure CertMagic can write to it!

The notion of a "cluster" or "fleet" of instances that may be serving the same site and sharing certificates, etc, is tied to storage. Simply, any instances that use the same storage facilities are considered part of the cluster. So if you deploy 100 instances of CertMagic behind a load balancer, they are all part of the same cluster if they share the same storage configuration. Sharing storage could be mounting a shared folder, or implementing some other distributed storage system such as a database server or KV store.

The easiest way to change the storage being used is to set certmagic.DefaultStorage to a value that satisfies the Storage interface. Keep in mind that a valid Storage must be able to implement some operations atomically in order to provide locking and synchronization.

If you write a Storage implementation, please add it to the project wiki so people can find it!

Cache

All of the certificates in use are de-duplicated and cached in memory for optimal performance at handshake-time. This cache must be backed by persistent storage as described above.

Most applications will not need to interact with certificate caches directly. Usually, the closest you will come is to set the package-wide certmagic.DefaultStorage variable (before attempting to create any Configs). However, if your use case requires using different storage facilities for different Configs (that's highly unlikely and NOT recommended! Even Caddy doesn't get that crazy), you will need to call certmagic.NewCache() and pass in the storage you want to use, then get new Config structs with certmagic.NewWithCache() and pass in the cache.

Again, if you're needing to do this, you've probably over-complicated your application design.

FAQ

Can I use some of my own certificates while using CertMagic?

Yes, just call the relevant method on the Config to add your own certificate to the cache:

Keep in mind that unmanaged certificates are (obviously) not renewed for you, so you'll have to replace them when you do. However, OCSP stapling is performed even for unmanaged certificates that qualify.

Does CertMagic obtain SAN certificates?

Technically all certificates these days are SAN certificates because CommonName is deprecated. But if you're asking whether CertMagic issues and manages certificates with multiple SANs, the answer is no. But it does support serving them, if you provide your own.

How can I listen on ports 80 and 443? Do I have to run as root?

On Linux, you can use setcap to grant your binary the permission to bind low ports:

$ sudo setcap cap_net_bind_service=+ep /path/to/your/binary

and then you will not need to run with root privileges.

Contributing

We welcome your contributions! Please see our contributing guidelines for instructions.

Project History

CertMagic is the core of Caddy's advanced TLS automation code, extracted into a library. The underlying ACME client implementation is ACMEz. CertMagic's code was originally a central part of Caddy even before Let's Encrypt entered public beta in 2015.

In the years since then, Caddy's TLS automation techniques have been widely adopted, tried and tested in production, and served millions of sites and secured trillions of connections.

Now, CertMagic is the actual library used by Caddy. It's incredibly powerful and feature-rich, but also easy to use for simple Go programs: one line of code can enable fully-automated HTTPS applications with HTTP->HTTPS redirects.

Caddy is known for its robust HTTPS+ACME features. When ACME certificate authorities have had outages, in some cases Caddy was the only major client that didn't experience any downtime. Caddy can weather OCSP outages lasting days, or CA outages lasting weeks, without taking your sites offline.

Caddy was also the first to sport "on-demand" issuance technology, which obtains certificates during the first TLS handshake for an allowed SNI name.

Consequently, CertMagic brings all these (and more) features and capabilities right into your own Go programs.

You can watch a 2016 dotGo talk by the author of this library about using ACME to automate certificate management in Go programs:

Matthew Holt speaking at dotGo 2016 about ACME in Go

Credits and License

CertMagic is a project by Matthew Holt, who is the author; and various contributors, who are credited in the commit history of either CertMagic or Caddy.

CertMagic is licensed under Apache 2.0, an open source license. For convenience, its main points are summarized as follows (but this is no replacement for the actual license text):

  • The author owns the copyright to this code
  • Use, distribute, and modify the software freely
  • Private and internal use is allowed
  • License text and copyright notices must stay intact and be included with distributions
  • Any and all changes to the code must be documented
Owner
Caddy
The ultimate server: enterprise-ready, extensible, open source, and automatic HTTPS with a configuration API
Caddy
Comments
  • [Gandi] Wrong DNS TXT record causes unable to get certificate

    [Gandi] Wrong DNS TXT record causes unable to get certificate

    What version of the package are you using?

    CADDY_VERSION=v2.2.0 xcaddy build --with github.com/caddy-dns/gandi

    What are you trying to do?

    Use Caddy as usual, with HTTPS solves with dns-01 challenge

    What steps did you take?

    cloud.skynewz.dev, *.cloud.skynewz.dev
    tls [email protected] {
    	dns gandi {env.GANDI_API_TOKEN}
    }
    respond "Hello, world!"
    

    What did you expect to happen, and what actually happened instead?

    Caddy can get my certificate and alright ! Instead, something creates a TXT record that Caddy cannot solve and Caddy never start, lopping on resolving the challenge

    Please link to any related issues, pull requests, and/or discussion

    https://github.com/caddyserver/caddy/issues/3787

  • Is Certmagic suitable for my use case of adding subdomains pointed to a reverse proxy?

    Is Certmagic suitable for my use case of adding subdomains pointed to a reverse proxy?

    Apologies if this is a duplicate of "Does certmagic have any limitations?"

    Here's my use case:

    I have a "distributed" reverse proxy/ specialized cache hosted in a few locations across the world that points to an origin server.

    1, I'll ask my customers to point a subdomain (or main domain) to my SaaS (both A and AAAA records). This should satisfy the bolded requirement in the docs.

    1. For my implementation of certmagic, I'll use on Demand TLS with:
    • Use a custom decisionFunc that grabs the domain from my backend and verifies it's a customer of mine
    • Write some sort of fancy certmagic.DefaultStorage to either grab the certificate from storage, or fetch it from a REST endpoint.

    I'm 99% sure this will work for me, I'm just looking for verification.

    Also, does this enable HTTP2?

  • Support OverrideDomain is DNS01Solver

    Support OverrideDomain is DNS01Solver

    CNAME can be used to delegate answering the chanllenge to another DNS zone, in order to reduce the exposure of the DNS credential [1]. The solver already follows CNAME when checking propagation against the authoritative source, it should follow CNAME when setting the TXT record as well.

    [1] https://letsencrypt.org/docs/challenge-types/#dns-01-challenge

  • Wildcard certificate is not loaded from storage

    Wildcard certificate is not loaded from storage

    What version of the package are you using?

    github.com/caddyserver/certmagic v0.11.0

    What are you trying to do?

    I'm attempting to setup a cluster configuration where go process do the TLS termination and share a common storage. However (and I suspect the problem start here), the certificates creation is done in another process connected to the same storage, by calling config.ManageSync() and exiting.

    (On a side note, it'd be nice if a lower level function would be exposed to check/generate/renew certificates without the higher level stuff that ManageSync imply).

    What steps did you take?

    After making sure that a wildcard certificate exist for *.example.com, I did a HTTPS request to foo.example.com on one of the nodes connected to the common storage.

    Following what happen with a debugger, it seems to me that the in-memory cache is checked first. This fail as the node doesn't have any certificates at that point.

    As a fallback, the storage is then checked. However, only the exact match for foo.example.com is attempted.

    (Side note again, an OnDemand config needs to be created for that to work. However, I do not want to generate new certificates, only load them from the disk.)

    What did you expect to happen, and what actually happened instead?

    I expected my wildcard certificates to be loaded.

    How do you think this should be fixed?

    I would think the storage should be checked for both foo.example.com and *.example.com. Possibly in parallel.

    Bonus: What do you use CertMagic for, and do you find it useful?

    Because it's the only way I could find to solve my problem. That's no small feat.

  • Listing new DynamoDB storage adapter

    Listing new DynamoDB storage adapter

    Hi @mholt, I wrote a new DynamoDB storage adapter, but before I add it to the list of known adapters I wanted to ask if you'd look it over and provide any feedback/suggestions/issues you have with it.

    https://github.com/silinternational/certmagic-storage-dynamodb

    Thanks!

  • version 0.9.1 regression causes a systematic crash (rolled back to 0.8.3 to fix)

    version 0.9.1 regression causes a systematic crash (rolled back to 0.8.3 to fix)

    What version of the package are you using?

    0.9.1

    What are you trying to do?

    We use successfully Certmagic in production since 10 months.

    		certmagic.Default.Agreed = true
    		certmagic.Default.Email = s.Email
    		certmagic.Default.DisableHTTPChallenge = true
    
    		cfg := certmagic.NewDefault()
    		tlsConfig := cfg.TLSConfig()
    		err := cfg.ManageSync(s.Hosts)
    		if err != nil {
    			return err
    		}
    

    What steps did you take?

    We just have updated to certmagic 0.9.0 (crashes happened regularly) With certmagic 0.9.1 panic is systematic So we have rolled back to 0.8.3 that is stable.

    2020/01/12 08:44:21 [INFO][cache:0xc000189720] Started certificate maintenance routine
    2020/01/12 08:44:22 [INFO][0.tplst.com] Renew certificate
    2020/01/12 08:44:22 [INFO][0.tplst.com] Renew: Waiting on rate limiter...
    2020/01/12 08:44:22 [INFO][0.tplst.com] Renew: Done waiting
    2020/01/12 08:44:22 [INFO] [0.tplst.com] acme: Trying renewal with 710 hours remaining
    2020/01/12 08:44:22 [INFO] [0.tplst.com] acme: Obtaining bundled SAN certificate
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x7667a5]
    
    goroutine 1 [running]:
    time.(*Timer).Stop(...)
    	/usr/local/go/src/time/sleep.go:74
    github.com/cenkalti/backoff/v3.(*defaultTimer).Stop(0xc00059c120)
    	/home/tplst/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/timer.go:32 +0x25
    github.com/cenkalti/backoff/v3.RetryNotifyWithTimer.func1(0xc1e9a0, 0xc00059c120)
    	/home/tplst/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:45 +0x31
    github.com/cenkalti/backoff/v3.RetryNotifyWithTimer(0xc000498420, 0x7f1daeca15e8, 0xc0004f7cc0, 0x0, 0xc1e9a0, 0xc00059c120, 0x0, 0x0)
    	/home/tplst/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:53 +0x34d
    github.com/cenkalti/backoff/v3.RetryNotify(...)
    	/home/tplst/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:31
    github.com/cenkalti/backoff/v3.Retry(...)
    	/home/tplst/go/pkg/mod/github.com/cenkalti/backoff/[email protected]/retry.go:25
    github.com/go-acme/lego/v3/acme/api.(*Core).retrievablePost(0xc0003e6000, 0xc0006f6500, 0x33, 0xc000730e40, 0x36, 0x40, 0xa09320, 0xc0003b63c0, 0xc000144610, 0xc0003b6460, ...)
    	/home/tplst/go/pkg/mod/github.com/go-acme/lego/[email protected]/acme/api/api.go:107 +0x210
    github.com/go-acme/lego/v3/acme/api.(*Core).post(0xc0003e6000, 0xc0006f6500, 0x33, 0xaebde0, 0xc0003b6460, 0xa09320, 0xc0003b63c0, 0x1, 0x200000003, 0xc000000180)
    	/home/tplst/go/pkg/mod/github.com/go-acme/lego/[email protected]/acme/api/api.go:70 +0xf5
    github.com/go-acme/lego/v3/acme/api.(*OrderService).New(0xc0003e60c0, 0xc0006db9c0, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    	/home/tplst/go/pkg/mod/github.com/go-acme/lego/[email protected]/acme/api/order.go:22 +0x240
    github.com/go-acme/lego/v3/certificate.(*Certifier).Obtain(0xc0006e5200, 0xc0006db9b0, 0x1, 0x1, 0x1, 0xabacc0, 0xc000738e40, 0x0, 0xc000086958, 0x0, ...)
    	/home/tplst/go/pkg/mod/github.com/go-acme/lego/[email protected]/certificate/certificates.go:102 +0x1ec
    github.com/go-acme/lego/v3/certificate.(*Certifier).Renew(0xc0006e5200, 0xc00072e250, 0xb, 0xc000086180, 0x53, 0xc0000861e0, 0x53, 0xc00074a000, 0xe3, 0x2e3, ...)
    	/home/tplst/go/pkg/mod/github.com/go-acme/lego/[email protected]/certificate/certificates.go:384 +0x427
    github.com/mholt/certmagic.(*acmeClient).tryRenew(0xc0006e5230, 0xc00072e250, 0xb, 0xc000086180, 0x53, 0xc0000861e0, 0x53, 0xc00074a000, 0xe3, 0x2e3, ...)
    	/home/tplst/go/pkg/mod/github.com/mholt/[email protected]/client.go:419 +0x80
    github.com/mholt/certmagic.(*acmeClient).Renew(0xc0006e5230, 0xc214a0, 0xc000026158, 0x7fff3ff07516, 0xb, 0x0, 0x0)
    	/home/tplst/go/pkg/mod/github.com/mholt/[email protected]/client.go:394 +0x624
    github.com/mholt/certmagic.(*Config).RenewCert(0xc0003e60f0, 0xc214a0, 0xc000026158, 0x7fff3ff07516, 0xb, 0xc0000b2400, 0x20f, 0x40f)
    	/home/tplst/go/pkg/mod/github.com/mholt/[email protected]/config.go:478 +0x183
    github.com/mholt/certmagic.(*Config).manageOne(0xc0003e60f0, 0xc214a0, 0xc000026158, 0x7fff3ff07516, 0xb, 0x2, 0x8)
    	/home/tplst/go/pkg/mod/github.com/mholt/[email protected]/config.go:426 +0x4f1
    github.com/mholt/certmagic.(*Config).manageAll(0xc0003e60f0, 0x0, 0x0, 0xc0000ab320, 0x12, 0x12, 0x7f1daeced000, 0x0, 0xc000459180)
    	/home/tplst/go/pkg/mod/github.com/mholt/[email protected]/config.go:394 +0x1d1
    github.com/mholt/certmagic.(*Config).ManageSync(...)
    	/home/tplst/go/pkg/mod/github.com/mholt/[email protected]/config.go:320
    gitlab.tplst.com/theplaylist/tplst/pkg/com.(*HttpServer).Serve(0xc000499c48, 0xc139c0, 0xc000093a00, 0x0, 0xc0000ab320)
    	/home/tplst/src/pkg/com/http_server.go:30 +0xd5
    gitlab.tplst.com/theplaylist/tplst/pkg/cmds.(*Tplst).start(0xc000093a00, 0xc000093a00, 0x7fff3ff0750f)
    	/home/tplst/src/pkg/cmds/tplst.go:158 +0xbe7
    gitlab.tplst.com/theplaylist/tplst/pkg/cmds.NewTplst(...)
    	/home/tplst/src/pkg/cmds/tplst.go:60
    main.main()
    	/home/tplst/src/tplst.go:85 +0x45e
    

    Is "github.com/cenkalti/backoff" really required?

  • Simpler Storage API

    Simpler Storage API

    It's probably too late to change the API now, so this is more of a question about why the API is designed the way it is than anything.

    What would you like to have changed?

    Remove Locker from the Storage interface.

    Why is this feature a useful, necessary, and/or important addition to this project?

    TryLock is hard to implement correctly and the implementation doesn't seem to make use of it, because it just waits anyway.

    What alternatives are there, or what are you doing in the meantime to work around the lack of this feature?

    The ideal interface would be Storage on its own without Locker, then all methods would be required to be safe for concurrent use. Apart from being more idiomatic, it makes things a lot simpler because the actual storage implementation has a better idea about how to do locking than CertMagic.

    The main motivation was to implement a Storage in SQLite (or other databases) but I was blocked by the requirement that TryLock being non-blocking. Instead of implementing the equivalent of mu.Lock(), it seemed I'd have to introduce a whole bunch of state that will probably be buggy for unknown reasons. Even FileStorage.TryLock makes me nervous because at a quick glance, it doesn't seem to handle cleanup so if the process crashes, everything is probably borked.

  • Wildcard domains (*.example.com)

    Wildcard domains (*.example.com)

    Would like to have wildcard domain support, right now certificates are issued for every subdomain i.e 1234.example.com 43443.example.com have separate certificates.

    We should create 1 certificate with *.example.com, and use it instead for every subdomain.

    This would be helpful for services which serve subdomains.

    Can start working on this if i have your ok @mholt

  • Subject Alternative Names (SAN) Support

    Subject Alternative Names (SAN) Support

    Hey,

    I'm using your library for a long time now. It's really nice and magic :p

    Recently I encountered the need to have Subject Alternative Names in certificates managed by certmagic. So I made a PR to add this feature.

    I did not want to break the API to have backward compatibility. So I added a new Manager interface in order to keep old code working.

  • Replace TryLock and Wait with Lock, and check for idempotency

    Replace TryLock and Wait with Lock, and check for idempotency

    This replaces TryLock and Wait with a single Lock function, and we check after obtaining a lock to make sure Obtain and Renew remain idempotent.

    See issue #5.

    /cc @DisposaBoy - please help test and review!

  • unexpected state: ready - possible to pass CA certificate?

    unexpected state: ready - possible to pass CA certificate?

    What is your question?

    I am trying to use certmagic with a private acme compatible CA. Currently I am struggling to get a certificate issued with the following command:

    root@host ~ # ./acme-test 
    2021/12/14 15:45:31 hostname.net: obtaining certificate: hostname.net] Obtain: hostname.net] finalizing order https://acme.myca.net:443/acme/ws/Acme.svc/order/_-1n546ZnCna12d:  unexpected state: ready - order already finalized (ca=https://acme.myca.net/acme/ws/Acme.svc/InternalWebserverACME/directory)
    

    I have worked with this CA using several other acme clients like cert-manager and acme.sh. Those clients worked fine, although I needed to provide them with the certificate of the private ACME CA. Is there a way to pass the certificate to certmagic?

    What have you already tried?

    My code so far for testing:

    package main
    
    import (
    	"log"
    	"net/http"
    
    	"github.com/caddyserver/certmagic"
    )
    
    func main() {
    	domains := []string{"hostname.net"}
    	CA := "https://acme.myca.ne/acme/ws/Acme.svc/InternalWebserverACME/directory"
    	Email := "[email protected]"
    
    	certmagic.DefaultACME.Agreed = true
    	certmagic.DefaultACME.Email = Email
    	certmagic.DefaultACME.CA = CA
    
    	handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    		w.Write([]byte("Hello, encrypted world!"))
    	})
    	if err := certmagic.HTTPS(domains, handler); err != nil {
    		log.Fatal(err)
    	}
    }
    

    Edit: FYI: some of the values are changed for security reasons.

  • Question: Fencing tokens for storage

    Question: Fencing tokens for storage

    What is your question?

    When implementing a storage backend the comment describing Lock() mentions the use of fencing tokens: https://github.com/caddyserver/certmagic/blob/9826a4c3549441ba6dddcfc5d561f097feb44e2a/storage.go#L108-L112 The way I understand this is that once you have the lock further interaction with the backend while holding the lock, like writing a file, should then include such a token so that the write can fail if the token is out of date.

    What I am not understanding is this: given that Lock() takes a name, potentially meaning there could be multiple different locks held for different operations at the same time, how would I map a Store() request with the appropriate lock so that it can access the correct fencing token?

    What have you already tried?

    I have investigated the Store() call and from what I can tell the only information available to it is the key name and data that should be written.

  • Request support for more dynamic mTLS client cert/csr management

    Request support for more dynamic mTLS client cert/csr management

    What would you like to have changed?

    An interface to produce/configure the mTLS client CSR, or possibly allow the existing one to ignore it altogether.

    Ideally, I'd like to be able to issue a custom CSR based on the information about the server and destination backend. Bonus points for being able to decide the details of the client cert during the TLS negotiation at the CertificateRequest phase (caddy server/module, server offered CAs, configured san, etc).

    Why is this feature a useful, necessary, and/or important addition to this project?

    Right now when using mTLS only the common name of the certificate can be set and it must be hard-coded (even replacers aren't supported). In a TLS negotiation a server may offer support for multiple CAs (which means different possible valid client certs we might chose from) or a client may want to use different client certs depending on destination backend. In effect, the reverse of a just-in-time server cert issuance by leveraging the CertificateRequest step of the TLS negotiation.

    I am implementing Vault PKI issuance support in my application, but Vault is somewhat picky about the cert request details and I'd like to be able to more fully specify the details of the request dynamically (or just use the REST api pki/issue/$thing that gives me back a pkey instead of the sign api pki/sign/$thing which uses a CSR as an input). Unfortunately, Caddy/Certmagic's interface here don't let me manipulate (or outright ignore) the internally produced CSR.

    What alternatives are there, or what are you doing in the meantime to work around the lack of this feature?

    If I attempt to be sneaky about it (say by making my module also a KeyIssuer and then caching the key internally in a pubkey mapping) the cache storage engine gets confused. There might be a bug there in fact if the subject in the CSR when it's being stored != the subject that's actually in the cert. It attempts to read the wrong path on disk and fails. Regardless, it's a bit of a hack to take this approach.

    Please link to any relevant issues, pull requests, or other discussions.

    https://ldapwiki.com/wiki/TLS%20Full%20Handshake https://www.vaultproject.io/api-docs/secret/pki#generate-certificate-and-key https://www.vaultproject.io/api-docs/secret/pki#sign-certificate

    BTW, thank you! The caddyserevr project's extensible design is really something amazing.

  • Storage check -- can it be removed?

    Storage check -- can it be removed?

    https://github.com/caddyserver/certmagic/blob/76f61c2947a20d86ca37669dbdc0ed7a96fc6c5f/config.go#L456-L461

    Depending on the storage this check can be quite expensive (in our case a remote KV store), and it happens in a rather "hot" area: Right when setting up a TLS connection. If we need to get a certificate from a CA this doesn't matter too much in the grand scheme of things, but if our storage has one we triple our roundtrips (1 to get the cert information from storage, and 2 for a write/read from storage to check it).

    I think this check might have some value, but not here:

    • If the storage is generally trustworthy, it should be checked before it gets configured for certmagic
    • If the storage is generally untrustworthy and needs regular checks, it should be part of that specific storage

    So, I'd propose to just drop these lines here, and possibly move the implementation of checkStorage as an example into documentation. WDYT?

  • Fix advanced cache initialization in README

    Fix advanced cache initialization in README

    Hi!

    When initializing a Cache with NewCache it's easy to miss that a cache must be attached to the configuration in CacheOptions.GetConfigForCert and is not set by the cache itself. If a Config with certCache == nil is used, it will cause panics at runtime (not immediately, so it's not so easy to detect the cause of the panics).

    I think the proposed change to the README will address this issue and hopefully prevent other users from encountering the same problems I did, as they where not trivial to debug.

    I think a good follow up discussion and possibly PR would be to look into either:

    • Returning an error if certCache == nil, so that the error can be more easily traced to its origin.
    • Filling in the certCache field if it's nil, so that the error is not possible to begin with.

    I think this can be solved in Cache.getConfig.

    It's not clear to me why the cache must be set in the Config returned by GetConfigForCert as it doesn't allow a different cache to be used in the first place (at least I think so). Since it can check whether the correct cache is used, can it simply not use the correct cache? I've not looked to deep into this, I thought it better to ask first.

  • Question: how to use on demand with managed certificates

    Question: how to use on demand with managed certificates

    What is your question?

    Hi 👋

    We have the following use case:

    • We need to manage a wildcard certificate: *.example.org
    • We need to manage unknown domains domain1.com, domain2.com etc...

    How should we set up the server? We're using only certmagic and no Caddy Server.

    What have you already tried?

    We tried something similar, it works but I'm not sure if that's the right way to do:

    config := &certmagic.Config{}
    cache := certmagic.NewCache(certmagic.CacheOptions{ 
      GetConfigForCert: func(cert certmagic.Certificate) (*certmagic.Config, error) {
        return config, nil
      },
    }
    
    // Set also the the issuer
    // ...
    
    server := certmagic.New(cache, *config)
    server.ManageAsync(context.TODO(), []string{"*.example.org"});
    
    // And then we specify the on demand config:
    
    certmagic.Default.OnDemand = &certmagic.OnDemandConfig{
      DecisionFunc: func(name string) error {
        // compute if on demand domain certificate should be issued or not
        return nil
      },
    }
    
    certmagic.HTTPS([]string{}, handler);
    

    To my understanding, the DecisionFunc should not be called when a domain like this is requested: subdomain.example.org because we manage that async and it should be called only for non-managed domains.

    Are we on the right path?

    Also, when we do something like this instead of managing async:

    certmagic.Default.OnDemand = &certmagic.OnDemandConfig{
      DecisionFunc: func(name string) error {
        // compute if on demand domain certificate should be issued or not
        return nil
      },
    }
    
    certmagic.HTTPS([]string{"*.example.org"}, handler);
    

    and then visit subdomain.example.org it creates a certificate for subdomain.example.org and does not use the wildcard certificate. Why is that like that?

  • Data directory should be located in APPDATA on Windows

    Data directory should be located in APPDATA on Windows

    What would you like to have changed?

    On Windows, certmagic stores its data in %USERPROFILE%/.local/share by default at the moment, which should be located in %APPDATA% as Windows' convention.

    Why is this feature a useful, necessary, and/or important addition to this project?

    Following the operating system's convention gives users better understanding of the directory's function and eliminates unnecessary directory paths.

    What alternatives are there, or what are you doing in the meantime to work around the lack of this feature?

    Certmagic is used by Hysteria, which is considering

    provide an option in "acme" for customizing the certificate storage directory.

    Please link to any relevant issues, pull requests, or other discussions.

    My issue in Hysteria repo

Order TLS certificates using ACME TLS-ALPN-01

Order TLS certificates using ACME TLS-ALPN-01

Jan 4, 2023
:lock: acmetool, an automatic certificate acquisition tool for ACME (Let's Encrypt)
:lock: acmetool, an automatic certificate acquisition tool for ACME (Let's Encrypt)

acmetool is an easy-to-use command line tool for automatically acquiring certificates from ACME servers (such as Let's Encrypt). Designed to flexibly

Dec 29, 2022
Go package to embed the Mozilla Included CA Certificate List

rootcerts Package rootcerts provides an embedded copy of the Mozilla Included CA Certificate List, more specifically the PEM of Root Certificates in M

Oct 21, 2022
Retrieve SSL certificate information

cert Retrieve SSL certificate information from provided hostname. Why I just simply want to retrieve a website's SSL certificate information in my ter

Oct 5, 2021
Cloud IP address ranges lookup tool + DNS subdomain enumeration + Certificate Transparency
Cloud IP address ranges lookup tool + DNS subdomain enumeration + Certificate Transparency

Cloud edge Lookup an IP to find the cloud provider and other details based on the provider's published JSON data Cloud edge is a recon tool focused on

Dec 12, 2022
An opinionated helper for generating tls certificates

Certificates helper This is an opinionated helper for generating tls certificates. It outputs only in PEM format but this enables you easily generate

Dec 17, 2022
Driftwood is a tool that can enable you to lookup whether a private key is used for things like TLS or as a GitHub SSH key for a user.
Driftwood is a tool that can enable you to lookup whether a private key is used for things like TLS or as a GitHub SSH key for a user.

Driftwood is a tool that can enable you to lookup whether a private key is used for things like TLS or as a GitHub SSH key for a user. Drift

Dec 29, 2022
Automatic Linux privesc via exploitation of low-hanging fruit
Automatic Linux privesc via exploitation of low-hanging fruit

Traitor Automatically exploit low-hanging fruit to pop a root shell. Linux privilege escalation made easy! Traitor packages up a bunch of methods to e

Jan 1, 2023
Product Analytics, Business Intelligence, and Product Management in a fully self-contained box
Product Analytics, Business Intelligence, and Product Management in a fully self-contained box

Engauge Concept It's not pretty but it's functional. Track user interactions in your apps and products in real-time and see the corresponding stats in

Nov 17, 2021
A fully self-contained Nmap like parallel port scanning module in pure Golang that supports SYN-ACK (Silent Scans)

gomap What is gomap? Gomap is a fully self-contained nmap like module for Golang. Unlike other projects which provide nmap C bindings or rely on other

Dec 10, 2022
HTTP/HTTPS MITM proxy and recorder.
HTTP/HTTPS MITM proxy and recorder.

Hyperfox Hyperfox is a security auditing tool that proxies and records HTTP and HTTPS traffic between two points. Installation You can install the lat

Jan 9, 2023
erchive is a go program that compresses and encrypts files and entire directories into .zep files (encrypted zip files).

erchive/zep erchive is a go program that compresses and encrypts files and entire directories into .zep files (encrypted zip files). it compresses usi

May 16, 2022
Check and exploit log4j2 vulnerability with single Go program.
Check and exploit log4j2 vulnerability with single Go program.

log4j2-exp Check and exploit log4j2 vulnerability with single Go program. You don't need to install anything except develop it. It supports ldaps and

Nov 9, 2022
Check and exploit log4j2 vulnerability with single Go program.
Check and exploit log4j2 vulnerability with single Go program.

Log4Shell Check and exploit log4j2 vulnerability with single Go program. You don't need to install anything except develop it. It supports ldaps and h

Jan 6, 2023
✒ A self-hosted, cross-platform service to sign iOS apps using any CI as a builder
✒ A self-hosted, cross-platform service to sign iOS apps using any CI as a builder

iOS Signer Service A self-hosted, cross-platform service to sign iOS apps using any CI as a builder Introduction There are many reasons to install app

Jan 7, 2023
"I do" stops interactive command if there is any potential risky pattern

Description ido (I do) executes your shell command provided as its input, but it may wait for you to confirm when there is some potential risky patter

Jan 2, 2023
Barebones Go program to issue DDNS updates to Amazon Route 53 service.

Route53 DDNS Very simple DDNS using AWS Route 53 #/bin/bash # AWS_ACCESS_KEY_ID example (fake) export AWS_ACCESS_KEY_ID=KkRbWpoyqLHo69dvoskn # AWS_

May 17, 2021
Custom GPG pinentry program for macOS that allows using Touch ID for fetching the password from the macOS keychain.
Custom GPG pinentry program for macOS that allows using Touch ID for fetching the password from the macOS keychain.

pinentry-touchid Custom GPG pinentry program for macOS that allows using Touch ID for fetching the password from the macOS keychain. Macbook Pro devic

Jan 1, 2023
Program brute forcing the passphrase of a private key

Description This project provide a program brute forcing the passphrase of a private key (EC/RSA). ℹ️ This project was created to allow me to learn th

Oct 12, 2022