retryablehttp package provides a familiar HTTP client interface with automatic retries and exponential backoff.

go-retryablehttp

Build Status Go Documentation

The retryablehttp package provides a familiar HTTP client interface with automatic retries and exponential backoff. It is a thin wrapper over the standard net/http client library and exposes nearly the same public API. This makes retryablehttp very easy to drop into existing programs.

retryablehttp performs automatic retries under certain conditions. Mainly, if an error is returned by the client (connection errors, etc.), or if a 500-range response code is received (except 501), then a retry is invoked after a wait period. Otherwise, the response is returned and left to the caller to interpret.

The main difference from net/http is that requests which take a request body (POST/PUT et. al) can have the body provided in a number of ways (some more or less efficient) that allow "rewinding" the request body if the initial request fails so that the full request can be attempted again. See the godoc for more details.

Version 0.6.0 and before are compatible with Go prior to 1.12. From 0.6.1 onward, Go 1.12+ is required. From 0.6.7 onward, Go 1.13+ is required.

Example Use

Using this library should look almost identical to what you would do with net/http. The most simple example of a GET request is shown below:

resp, err := retryablehttp.Get("/foo")
if err != nil {
    panic(err)
}

The returned response object is an *http.Response, the same thing you would usually get from net/http. Had the request failed one or more times, the above call would block and retry with exponential backoff.

Getting a stdlib *http.Client with retries

It's possible to convert a *retryablehttp.Client directly to a *http.Client. This makes use of retryablehttp broadly applicable with minimal effort. Simply configure a *retryablehttp.Client as you wish, and then call StandardClient():

retryClient := retryablehttp.NewClient()
retryClient.RetryMax = 10

standardClient := retryClient.StandardClient() // *http.Client

For more usage and examples see the godoc.

Owner
HashiCorp
Consistent workflows to provision, secure, connect, and run any infrastructure for any application.
HashiCorp
Comments
  • Added support for HTTP 429 Retry-After as per Issue #99

    Added support for HTTP 429 Retry-After as per Issue #99

    This was added in DefaultPolicy and DefaultBackoff. The latter will examine the Retry-After response header (if existant) and do a backoff for the specified amount of time. Otherwise it will default to default backoff behaviour.

    This addresses Issue #99.

  • feat: allow access to LogHooks with nil Logger

    feat: allow access to LogHooks with nil Logger

    Summary

    • Allow RequestLogHook and ResponseLogHook to be accessed even if the client.Logger is nil
    • Add tests for ^

    Use Case

    We're using a logging package that doesn't conform to Logger interface, but still want access to values available through LogHooks. Specifically, we're using RequestLogHook to access the retry count and pass it to our own internal structured JSON logging package.

    However, setting Client.Logger = nil (in order to suppress logging to STDOUT) skips the block of code where RequestLogHook accesses the retry count (here). The presence of the default case in the switch here makes me think that this behavior (suppressed logging, but still able to access values in LogHooks) was meant to be supported.

    Considered Alternatives

    The alternative of disabling the logger with a noop logger or other similar approach has a small performance cost (see here, tho the benchmarks are probably out of date, the idea that it will cost something to have a noop logger is what is most important)

  • Fix potential data race, introduced by rand.NewSource

    Fix potential data race, introduced by rand.NewSource

    rand.NewSource doc clearly says: Unlike the default Source used by top-level functions, this source is not safe for concurrent use by multiple goroutines.

  • Drop request wrapper

    Drop request wrapper

    This fixes #22 in a different way to the current proposed fix (#21):

    • Do() can now take any http.Request as input. If the request body is not seekable then the contents of the request are read to a buffer, and bytes.Reader is used, which is seekable.
      • but if the request body is already seekable then the copy is not made
      • and in my experience almost all request bodies are already in-memory buffers created by marshaling json / xml / text / etc, so I don't think this is a big loss
      • and realistically, if the user passes in a request body that was not seekable, then under the old API the user just would have done the in-memory copy trick anyway
      • so it really does not seem like we're losing a lot here
    • drops the Request wrapper struct (converts it to a type alias for http.Request for backwards compatbility)
    • the NewRequest function is now just a direct passthrough to http.newRequest

    One of the nice things about this approach is that it allows users of this library to define interfaces that can match both http.Client and retryablehttp.Client, so e.g. that retryable HTTP requests can be dependency-injected into applications:

    type MyHttpClient interface {
      Do(req *http.Request) (*http.Response, error)
    }
    
    func doSomething(client MyHttpClient) { ... }
    
    func main() {
      if (shouldBeRetryable) {
        doSomething(&retryablehttp.Client{...})
      } else {
       doSomething(&http.Client{...})
      }
    }
    
  • Errors from original http.Client response are double-wrapped when using the RoundTripper

    Errors from original http.Client response are double-wrapped when using the RoundTripper

    When you're using a http.Client with a custom RoundTripper that actually facilitates the retries (it maintains its own internal http.Client object); any errors returned from the original http.Client are re-wrapped.

    For instance, trying to do a Get request that returns an error response, and using the PassthroughErrorHandler to return the real failure would result in an error like this:

    Get "http://this-url-does-not-exist.com/": Get "http://this-url-does-not-exist.com/": dial tcp: lookup this-url-does-not-exist.com: no such host
    

    This is basically an *url.Error object nested within another *url.Error's Err field.

    We can just .Unwrap() it to make sure that we only get what we need. Open to other suggestions though.

  • Why close all idle connections when retry check is not OK?

    Why close all idle connections when retry check is not OK?

    Hi, thanks for developing such wonderful library. I was trying to use it in my project and I have a question on below line.

    https://github.com/hashicorp/go-retryablehttp/blob/4af2e4b1970397d927d9e8d74c97e8659c0350c3/client.go#L575-L581

    May I know why need to do c.HTTPClient.CloseIdleConnections() when do not need retry? I found this will send a FIN to server side and start to close connection. But In my understanding, the connection can be still reused. Actually in my project, I found the connection number was keep growing after my service starts and accepts API requests. Then I removed this line in my project, the function is working and connections stop to keep growing.

  • Library closes all connections, makes reuse impossible

    Library closes all connections, makes reuse impossible

    As a followup to: https://github.com/hashicorp/go-retryablehttp/pull/57#pullrequestreview-234625048

    If I'm reading this correctly, the library currently seems to close all happy-case connections, which makes it unsuitable for use in a micro-service architecture where many requests go to the same few hosts. I'm currently observing this behavior in one of my services. This is not just a connection-time issue, but in my case the closed requests eat up available sockets in TIME_WAIT state to the point of machine failure.

    @jefferai Could you please revisit this review request above with this consideration in mind? I'm happy to PR this is you concur it's an issue.

  • adds debug and error methods to logger interface

    adds debug and error methods to logger interface

    This change makes it easier to use a different log implementation and use the log level from these loggers. I left out other additional ones (Infof, etc) since they are not used here

  • Replace hclog.Logger with  LeveledLogger interface

    Replace hclog.Logger with LeveledLogger interface

    what was done

    • Changing the hclog.Logger interface to a smaller one that will be easier to get implemented from other logger libraries. As we are just reducing the required methods on the interface this will not break the API.

    alternative:

    As proposed on #74 we can also break the API by changing the existing Logger interface.

    Closes #74

  • Propogate Do() HTTP error up when retries are exceeded

    Propogate Do() HTTP error up when retries are exceeded

    Add propagation of the HTTP request/response error up the call stack when the number of retries are exceeded. Because the ErrorHandler does not receive a copy of the request, it's impossible to use the ErrorHandler to generate an enhanced version of the default Retries Exceeded error message.

    This change takes the non-API breaking approach to solving this, by including the error string in the returned error (if an error has occured).

    Fixes #69

  • Fix race condition in rand.Source

    Fix race condition in rand.Source

    This reverts #33, fixing the data race, and removes the excess allocations by using the global random source which is safe for concurrent use.

    An alternative PR to #35

  • Update error log

    Update error log

    Change ERR to ERROR for consistency in our libraries.

    To Do:

    • Find out context for the different loggers. When running a terraform apply I saw the following log output from this library:
    2022-12-23T18:19:35.431-0700 [INFO]  CLI command args: []string{"apply"}
    2022/12/23 18:19:35 [DEBUG] GET https://app.terraform.io/.well-known/terraform.json
    

    The DEBUG prints even if TF_LOG is set to info (which I did in this case). Also would be nice to have consistent date/time stamps.

  • Go 1.20: fix up retry certificate retry  after Go CL

    Go 1.20: fix up retry certificate retry after Go CL "crypto/tls: add CertificateVerificationError to tls handshake"

    Certificate retry inhibiting would be broken in Go 1.20.

    The upstream CL http://go.dev/cl/449336 changed the error type returned when there's a certificate verification failure. The following line no longer hits:

    https://github.com/hashicorp/go-retryablehttp/blob/493aa4cf372e47ec00228558d9d91a9517b045a5/client.go#L476-L478

    I believe it would resume working (and also work for older Go versions) if this was changed to:

    			// Don't retry if the error was due to TLS cert verification failure.
    -			if _, ok := v.Err.(x509.UnknownAuthorityError); ok {
    +			if _, ok := errors.As(v.Err, &x509.UnknownAuthorityError{}); ok {
    				return false, nil
    			}
    
  • Feature request: give up early on 505, 506, 510, and 511

    Feature request: give up early on 505, 506, 510, and 511

    In addition to 501, these errors seem permanent as well:

    http.StatusHTTPVersionNotSupported       // 505 HTTP Version Not Supported
    http.StatusVariantAlsoNegotiates         // 506 Variant Also Negotiates
    http.StatusNotExtended                   // 510 Not Extended
    http.StatusNetworkAuthenticationRequired // 511 Network Authentication Required
    

    Perhaps we should skip the retry logic when we see one of them?

    PS: I'm not sure about http.StatusLoopDetected (508 Loop Detected). Maybe we should include that as well.

  • [COMPLIANCE] Update MPL-2.0 LICENSE

    [COMPLIANCE] Update MPL-2.0 LICENSE

    Hi there 👋

    This PR was auto-generated as part of an internal review of public repositories that are not in compliance with HashiCorp's licensing standards.

    Frequently Asked Questions

    Why am I getting this PR? This pull request was created because one or more of the following criteria was found:
    • This repo did not previously have a LICENSE file
    • A LICENSE file was present, but had a non-conforming name (e.g., license.txt)
    • A LICENSE file was present, but was missing an appropriate copyright statement

    More info is available in the RFC

    How do you determine the copyright date? The copyright date given in this PR is supposed to be the year the repository or project was created (whichever is older). If you believe the copyright date given in this PR is not valid, please reach out to:

    #proj-software-copyright

    I don't think this repo should be licensed under the terms of the Mozilla Public License 2.0. Who should I reach out to? If you believe this repository should not use an MPL 2.0 License, please reach out to [email protected]. Exemptions are considered on a case-by-case basis, but common reasons include if the project is co-managed by another entity that requires differing license terms, or if the project is part of an ecosystem that commonly uses a different license type (e.g., MIT or Apache 2.0).

    Please approve and merge this PR in a timely manner to keep this source code compliant with our OSS license agreement. If you have any questions or feedback, reach out to #proj-software-copyright.

    Thank you!


    Made with :heart: @HashiCorp

  • CodeQL Analysis throwing warning

    CodeQL Analysis throwing warning

    For lines

    https://github.com/hashicorp/go-retryablehttp/blob/493aa4cf372e47ec00228558d9d91a9517b045a5/client.go#L589

    and

    https://github.com/hashicorp/go-retryablehttp/blob/493aa4cf372e47ec00228558d9d91a9517b045a5/client.go#L646

    and

    https://github.com/hashicorp/go-retryablehttp/blob/493aa4cf372e47ec00228558d9d91a9517b045a5/client.go#L690

    CodeQL analysis is throwing following error

    Log entry depends on a user-provided value.
    
Go library that makes it easy to add automatic retries to your projects, including support for context.Context.

go-retry Go library that makes it easy to add automatic retries to your projects, including support for context.Context. Example with context.Context

Aug 15, 2022
Httpx - a fast and multi-purpose HTTP toolkit allow to run multiple probers using retryablehttp library
Httpx - a fast and multi-purpose HTTP toolkit allow to run multiple probers using retryablehttp library

httpx is a fast and multi-purpose HTTP toolkit allow to run multiple probers using retryablehttp library, it is designed to maintain the result reliability with increased threads.

Feb 3, 2022
Retry - Efficient for-loop retries in Go

retry Package retry implements an efficient loop-based retry mechanism that allo

Aug 23, 2022
fhttp is a fork of net/http that provides an array of features pertaining to the fingerprint of the golang http client.

fhttp The f stands for flex. fhttp is a fork of net/http that provides an array of features pertaining to the fingerprint of the golang http client. T

Jan 1, 2023
A nicer interface for golang stdlib HTTP client

rq A nicer interface for golang stdlib HTTP client Documents rq: here client: here jar: here Why? Because golang HTTP client is a pain in the a... Fea

Dec 12, 2022
Http client call for golang http api calls

httpclient-call-go This library is used to make http calls to different API services Install Package go get

Oct 7, 2022
Fast HTTP package for Go. Tuned for high performance. Zero memory allocations in hot paths. Up to 10x faster than net/http
Fast HTTP package for Go. Tuned for high performance. Zero memory allocations in hot paths. Up to 10x faster than net/http

fasthttp Fast HTTP implementation for Go. Currently fasthttp is successfully used by VertaMedia in a production serving up to 200K rps from more than

Jan 2, 2023
Simple HTTP package that wraps net/http

Simple HTTP package that wraps net/http

Jan 17, 2022
Simple HTTP and REST client library for Go

Resty Simple HTTP and REST client library for Go (inspired by Ruby rest-client) Features section describes in detail about Resty capabilities Resty Co

Jan 1, 2023
A Go HTTP client library for creating and sending API requests
A Go HTTP client library for creating and sending API requests

Sling Sling is a Go HTTP client library for creating and sending API requests. Slings store HTTP Request properties to simplify sending requests and d

Jan 7, 2023
Replacement of ApacheBench(ab), support for transactional requests, support for command line and package references to HTTP stress testing tool.

stress stress is an HTTP stress testing tool. Through this tool, you can do a stress test on the HTTP service and get detailed test results. It is ins

Aug 23, 2022
Speak HTTP like a local. (the simple, intuitive HTTP console, golang version)

http-gonsole This is the Go port of the http-console. Speak HTTP like a local Talking to an HTTP server with curl can be fun, but most of the time it'

Jul 14, 2021
NATS HTTP Round Tripper - This is a Golang http.RoundTripper that uses NATS as a transport.

This is a Golang http.RoundTripper that uses NATS as a transport. Included is a http.RoundTripper for clients, a server that uses normal HTTP Handlers and any existing http handler mux and a Caddy Server transport.

Dec 6, 2022
Http-conection - A simple example of how to establish a HTTP connection using Golang

A simple example of how to establish a HTTP connection using Golang

Feb 1, 2022
Full-featured, plugin-driven, extensible HTTP client toolkit for Go

gentleman Full-featured, plugin-driven, middleware-oriented toolkit to easily create rich, versatile and composable HTTP clients in Go. gentleman embr

Dec 23, 2022
An enhanced http client for Golang
An enhanced http client for Golang

go-http-client An enhanced http client for Golang Documentation on go.dev ?? This package provides you a http client package for your http requests. Y

Dec 23, 2022
An enhanced HTTP client for Go
An enhanced HTTP client for Go

Heimdall Description Installation Usage Making a simple GET request Creating a hystrix-like circuit breaker Creating a hystrix-like circuit breaker wi

Jan 9, 2023
Enriches the standard go http client with retry functionality.

httpRetry Enriches the standard go http client with retry functionality using a wrapper around the Roundtripper interface. The advantage of this libra

Dec 10, 2022
http client for golang
http client for golang

Request HTTP client for golang, Inspired by Javascript-axios Python-request. If you have experience about axios or requests, you will love it. No 3rd

Dec 18, 2022