Record and replay your HTTP interactions for fast, deterministic and accurate tests

go-vcr

Build Status GoDoc Go Report Card codecov

go-vcr simplifies testing by recording your HTTP interactions and replaying them in future runs in order to provide fast, deterministic and accurate testing of your code.

go-vcr was inspired by the VCR library for Ruby.

Installation

Install go-vcr by executing the command below:

$ go get github.com/dnaeon/go-vcr/recorder

Usage

Here is a simple example of recording and replaying etcd HTTP interactions.

You can find other examples in the example directory of this repository as well.

package main

import (
	"log"
	"time"

	"github.com/dnaeon/go-vcr/recorder"

	"github.com/coreos/etcd/client"
	"golang.org/x/net/context"
)

func main() {
	// Start our recorder
	r, err := recorder.New("fixtures/etcd")
	if err != nil {
		log.Fatal(err)
	}
	defer r.Stop() // Make sure recorder is stopped once done with it

	// Create an etcd configuration using our transport
	cfg := client.Config{
		Endpoints:               []string{"http://127.0.0.1:2379"},
		HeaderTimeoutPerRequest: time.Second,
		Transport:               r, // Inject as transport!
	}

	// Create an etcd client using the above configuration
	c, err := client.New(cfg)
	if err != nil {
		log.Fatalf("Failed to create etcd client: %s", err)
	}

	// Get an example key from etcd
	etcdKey := "/foo"
	kapi := client.NewKeysAPI(c)
	resp, err := kapi.Get(context.Background(), etcdKey, nil)

	if err != nil {
		log.Fatalf("Failed to get etcd key %s: %s", etcdKey, err)
	}

	log.Printf("Successfully retrieved etcd key %s: %s", etcdKey, resp.Node.Value)
}

Custom Request Matching

During replay mode, You can customize the way incoming requests are matched against the recorded request/response pairs by defining a Matcher function. For example, the following matcher will match on method, URL and body:

r, err := recorder.New("fixtures/matchers")
if err != nil {
	log.Fatal(err)
}
defer r.Stop() // Make sure recorder is stopped once done with it

r.SetMatcher(func(r *http.Request, i cassette.Request) bool {
	if r.Body == nil {
		return cassette.DefaultMatcher(r, i)
	}
	var b bytes.Buffer
	if _, err := b.ReadFrom(r.Body); err != nil {
		return false
	}
	r.Body = ioutil.NopCloser(&b)
	return cassette.DefaultMatcher(r, i) && (b.String() == "" || b.String() == i.Body)
})

Protecting Sensitive Data

You often provide sensitive data, such as API credentials, when making requests against a service. By default, this data will be stored in the recorded data but you probably don't want this. Removing or replacing data before it is stored can be done by adding one or more Filters to your Recorder. Here is an example that removes the Authorization header from all requests:

r, err := recorder.New("fixtures/filters")
if err != nil {
	log.Fatal(err)
}
defer r.Stop() // Make sure recorder is stopped once done with it

// Add a filter which removes Authorization headers from all requests:
r.AddFilter(func(i *cassette.Interaction) error {
    delete(i.Request.Headers, "Authorization")
    return nil
})

Sensitive data in responses

Filters added using *Recorder.AddFilter are applied within VCR's custom http.Transport. This means that if you edit a response in such a filter then subsequent test code will see the edited response. This may not be desirable in all cases. For instance, if a response body contains an OAuth access token that is needed for subsequent requests, then redact the access token in SaveFilter will result in authorization failures.

Another way to edit recorded interactions is to use *Recorder.AddSaveFilter. Filters added with this method are applied just before interactions are saved when *Recorder.Stop is called.

r, err := recorder.New("fixtures/filters")
if err != nil {
	log.Fatal(err)
}
defer r.Stop() // Make sure recorder is stopped once done with it

// Your test code will continue to see the real access token and
// it is redacted before the recorded interactions are saved     
r.AddSaveFilter(func(i *cassette.Interaction) error {
    if strings.Contains(i.URL, "/oauth/token") {
        i.Response.Body = `{"access_token": "[REDACTED]"}`
    }

    return nil
})

Passing Through Requests

Sometimes you want to allow specific requests to pass through to the remote server without recording anything.

Globally, you can use ModeDisabled for this, but if you want to disable the recorder for individual requests, you can add Passthrough functions to the recorder. The function takes a pointer to the original request, and returns a boolean, indicating if the request should pass through to the remote server.

Here's an example to pass through requests to a specific endpoint:

// Pass through the request to the remote server if the path matches "/login".
r.AddPassthrough(func(req *http.Request) bool {
    return req.URL.Path == "/login"
})

License

go-vcr is Open Source and licensed under the BSD License

Owner
Marin Atanasov Nikolov
Marin Atanasov Nikolov
Comments
  • v2.1.0 NewAsMode fails with ModeReplaying and missing cassette file

    v2.1.0 NewAsMode fails with ModeReplaying and missing cassette file

    The latest minor version update contained a breaking change in the NewAsMode function.

    https://github.com/dnaeon/go-vcr/blob/6bd2f2d37e012291a856c69d6147a5f3cef0d218/v2/recorder/recorder.go#L193-L198

    In the previous version we were using (v.2.0.1), this function just created an empty cassette if there was no cassette file found with the given name. With v2.1.0 this behavior changed and calling NewAsMode using ModeReplaying in combination with a non-existing cassette filename returns an error. Especially in test scenarios where the recorder is just setup in a SetupTest method for all tests, independent of whether all test methods execute an outgoing http call, this is an undesired behavior.

    Could we get this function backwards compatible by introducing a flag which defines whether a missing cassette file should return an error?

    Thanks in advance!

  • feat: add `*Recorder.AddSaveFilter` to redact interactions before saving

    feat: add `*Recorder.AddSaveFilter` to redact interactions before saving

    Closes #54

    Adds a new type of filter that is applied to interactions just before they are saved. These filters are added using *Recorder.AddSaveFilter.

    Why do we need this?

    (the following explanation assumes VCR is in recording mode)

    Imagine a work with an API that authenticates and authorizes requests using OAuth client credentials. The first step when interacting with this API is to acquire an access token and then make the intended API requests:

    > POST /oauth/token
    client_id=my_sample_app&client_secret=super-secret-credentials
    
    < {"access_token": "390278941074fe1a1acd"}
    
    
    > POST /blog/1234/comments
    > Authorization: Bearer 390278941074fe1a1acd
    
    {"message": "cool blog posts"}
    
    < {"comment_id": "8043891dde"}
    

    If we use an AddFilter to redact to scrub the access_token like so:

    rec.AddFilter(func(i *cassette.Interaction) error {
    	i.Response.Body = `{"access_token": "[REDACTED]"}`
    	return nil
    })
    

    Then the second request will be made like so:

    > POST /blog/1234/comments
    > Authorization: Bearer [REDACTED]
    
    {"message": "cool blog posts"}
    
    < {"comment_id": "8043891dde"}
    

    This is because the response is mutated in the recorder's RoundTripper and the mutation is used to build the http.Response before being passed on to the caller.

    So this means we cannot interfere with the response at the point where the current filters are added.

    Solution

    To get around this problem, we allow VCR users to register another set of filters that are applied when interactions are being saved. At that point, it is safe to mutate recorded interactions.

  • Add a new mode: ModeRecordingOrReplaying

    Add a new mode: ModeRecordingOrReplaying

    I'd like to have a new mode: ModeRecordingOrReplaying. It will do the recording if cassette returns ErrInteractionNotFound. Otherwise it does replaying. It is useful to me. How about others?

  • add filter after reading the cassette

    add filter after reading the cassette

    sometime we wants to modify the recorded values dynamically.

    for example some 3rd party library generates a requestID on the HTTP header and verify it when it got the response back.

    for example maybe a ReplyFilter can be added to give the caller a chance to modify the request or response

  • Strip Authentication string from headers

    Strip Authentication string from headers

    Hello!

    I was wondering, is there any way of modifying what is being dumped to the recorded response? For example, in the original ruby vcr gem there is an option to filter_sensitive_data, but for go-vcr package I can't figure out a way how to do the same.

    Currently, I'm doing this as a quick workaround, but I don't think that it's the best way to do that:

    if copiedReq.Header["Authorization"] != nil {
      copiedReq.Header["Authorization"] = []string{"<FILTERED>"}
    }
    
    // Add interaction to cassette
    interaction := &cassette.Interaction{
      Request: cassette.Request{
        Headers: copiedReq.Header,
        ...
      },
      ...
    }
    

    Could anyone suggest how to do that better? Maybe we can introduce configurable Interaction struct which will be an optional parameter of a New?

  • Add DiscardOnSave field to Interaction type

    Add DiscardOnSave field to Interaction type

    The DiscardOnSave field may be set to true by hooks, in order to discard an interaction before saving it on disk.

    See issue #80 for some background info.

  • Save cassettes without any interactions

    Save cassettes without any interactions

    This is often required when writing table-driven tests where some of the test cases don't produce any interactions (like when validation fails) but the setup and test code is the same for every test case. In this case, on a replay these tests then fail because the cassette for the test cannot be found.

    Imagine this test code (I'll try to make it as succinct as possible):

    package example
    
    import (
    	"flag"
    	"testing"
    
    	"gopkg.in/dnaeon/go-vcr.v3/recorder"
    )
    
    var updateTestdata = flag.Bool("update", false, "update test data")
    
    func TestSomething(t *testing.T) {
    	type args struct {
    		service string
    	}
    	tests := []struct {
    		name string
    	}{
    		{
    			name: "test 1",
    		},
    		{
    			name: "test 2",
    		},
    	}
    	for _, tt := range tests {
    		t.Run(tt.name, func(t *testing.T) {
    			cassetteName := t.Name()
    			opts := &recorder.Options{
    				CassetteName: cassetteName,
    				Mode:         recorder.ModeReplayOnly,
    			}
    			if *updateTestdata {
    				opts.Mode = recorder.ModeRecordOnly
    			}
    
    			recorder, err := recorder.NewWithOptions(opts)
    			if err != nil {
    				t.Fatalf("creating recorder: %v", err)
    			}
    
    			// test code
    
    			if err := recorder.Stop(); err != nil {
    				t.Fatalf("stopping recorder: %v", err)
    			}
    		})
    	}
    }
    

    Imagine this were testing an HTTP API and some tests would fail validation and not do any calls to the API that's being tested, resulting in no interactions for that cassette.

    I have a Makefile target that runs this using go test -v ./... -updateTestdata which sets the mode to recorder.ModeRecordOnly. In CI the mode is then set to recorder.ModeReplayOnly. Because the cassette isn't there with the current code, it'll fail on those tests. I also decided against using a single cassette for the whole test function because that's brittle and will prevent us from adding more test cases in the middle and will result in one big cassette that can't be grasped easily.

    Dropping these three lines of code will write a file like this:

    ---
    version: 2
    interactions: []
    

    …which is perfectly good. Removing this code also doesn't break any tests.

    I know, of course, that this is a behavior change. I can also make it configurable but I decided against that for now.

  • Update gopkg.in/yaml.v2 from v2.2.1 to v2.2.8

    Update gopkg.in/yaml.v2 from v2.2.1 to v2.2.8

    • Updated yaml.v2 version in go.mod from v2.2.1 to v2.2.8
    • Executed go mod vendor to update content of vendor directory
    • Executed make test to run included unittests
  • Tests fail if filter modifies URL

    Tests fail if filter modifies URL

    Adding this filter to my recorder makes tests to fail.

    		r.AddFilter(func(_ *cassette.Interaction) error {
    			u, _ := url.Parse(i.Request.URL)
    			q := u.Query()
    			q.Set("access_key", "no-op")
    			u.RawQuery = q.Encode()
    			i.Request.URL = u.String()
    			return nil
    		})
    

    The service I'm building this library for insists on using the access key as a URL param since they've been doing that since launch nearly 10 years ago.

    Since I'm snapshot testing the output of the requests, I don't want the access key getting logged.

    But, modifying the interaction causes subsequent tests to fail.

    What can I do to fix this?

    Thanks!

  • honour request context even during replay

    honour request context even during replay

    The context.Context in http.Requests needs to be checked & honoured, even during a replay. Otherwise a change in tests or behaviour may be masked & not caught by tests.

  • allowing matcher to see the order of requests

    allowing matcher to see the order of requests

    i just hit a scenario where having the order of the requests is more useful than complex matcher.

    An easy change to the matcher function adding the position in the array of the requests, allow you to check that the order of the requests is correct and use a simple matcher.

    If you are interested in this I can arrange a pull request quite easly.

Rr-e2e-tests - Roadrunner end-to-end tests repository
Rr-e2e-tests - Roadrunner end-to-end tests repository

RoadRunner end-to-end plugins tests License: The MIT License (MIT). Please see L

Dec 15, 2022
Sql mock driver for golang to test database interactions

Sql driver mock for Golang sqlmock is a mock library implementing sql/driver. Which has one and only purpose - to simulate any sql driver behavior in

Dec 31, 2022
A Go implementation of Servirtium, a library that helps test interactions with APIs.

Servirtium is a server that serves as a man-in-the-middle: it processes incoming requests, forwards them to a destination API and writes the response into a Markdown file with a special format that is common across all of the implementations of the library.

Jun 16, 2022
Go-interactions - Easy slash commands for Arikawa

go-interactions A library that aims to make dealing with discord's slash command

May 26, 2022
Fortio load testing library, command line tool, advanced echo server and web UI in go (golang). Allows to specify a set query-per-second load and record latency histograms and other useful stats.
Fortio load testing library, command line tool, advanced echo server and web UI in go (golang). Allows to specify a set query-per-second load and record latency histograms and other useful stats.

Fortio Fortio (Φορτίο) started as, and is, Istio's load testing tool and now graduated to be its own project. Fortio is also used by, among others, Me

Jan 2, 2023
Extremely flexible golang deep comparison, extends the go testing package and tests HTTP APIs
Extremely flexible golang deep comparison, extends the go testing package and tests HTTP APIs

go-testdeep Extremely flexible golang deep comparison, extends the go testing package. Latest news Synopsis Description Installation Functions Availab

Dec 22, 2022
A simple and expressive HTTP server mocking library for end-to-end tests in Go.

mockhttp A simple and expressive HTTP server mocking library for end-to-end tests in Go. Installation go get -d github.com/americanas-go/mockhttp Exa

Dec 19, 2021
A next-generation testing tool. Orion provides a powerful DSL to write and automate your acceptance tests

Orion is born to change the way we implement our acceptance tests. It takes advantage of HCL from Hashicorp t o provide a simple DSL to write the acceptance tests.

Aug 31, 2022
Automatically update your Go tests

autogold - automatically update your Go tests autogold makes go test -update automatically update your Go tests (golden files and Go values in e.g. fo

Dec 25, 2022
Terratest is a Go library that makes it easier to write automated tests for your infrastructure code.

Terratest is a Go library that makes it easier to write automated tests for your infrastructure code. It provides a variety of helper functions and patterns for common infrastructure testing tasks,

Dec 30, 2022
Testing framework for Go. Allows writing self-documenting tests/specifications, and executes them concurrently and safely isolated. [UNMAINTAINED]

GoSpec GoSpec is a BDD-style testing framework for the Go programming language. It allows writing self-documenting tests/specs, and executes them in p

Nov 28, 2022
Robust framework for running complex workload scenarios in isolation, using Go; for integration, e2e tests, benchmarks and more! 💪

e2e Go Module providing robust framework for running complex workload scenarios in isolation, using Go and Docker. For integration, e2e tests, benchma

Jan 5, 2023
How we can run unit tests in parallel mode with failpoint injection taking effect and without injection race

This is a simple demo to show how we can run unit tests in parallel mode with failpoint injection taking effect and without injection race. The basic

Oct 31, 2021
Snapshot - snapshot provides a set of utility functions for creating and loading snapshot files for using snapshot tests.

Snapshot - snapshot provides a set of utility functions for creating and loading snapshot files for using snapshot tests.

Jan 27, 2022
Package for comparing Go values in tests

Package for equality of Go values This package is intended to be a more powerful and safer alternative to reflect.DeepEqual for comparing whether two

Dec 29, 2022
Go testing in the browser. Integrates with `go test`. Write behavioral tests in Go.
Go testing in the browser. Integrates with `go test`. Write behavioral tests in Go.

GoConvey is awesome Go testing Welcome to GoConvey, a yummy Go testing tool for gophers. Works with go test. Use it in the terminal or browser accordi

Dec 30, 2022
Ruby on Rails like test fixtures for Go. Write tests against a real database

testfixtures Warning: this package will wipe the database data before loading the fixtures! It is supposed to be used on a test database. Please, doub

Jan 8, 2023
A simple `fs.FS` implementation to be used inside tests.

testfs A simple fs.FS which is contained in a test (using testing.TB's TempDir()) and with a few helper methods. PS: This lib only works on Go 1.16+.

Mar 3, 2022
Go testing in the browser. Integrates with `go test`. Write behavioral tests in Go.
Go testing in the browser. Integrates with `go test`. Write behavioral tests in Go.

GoConvey is awesome Go testing Welcome to GoConvey, a yummy Go testing tool for gophers. Works with go test. Use it in the terminal or browser accordi

Dec 30, 2022