'go test' runner with output optimized for humans, JUnit XML for CI integration, and a summary of the test results.

gotestsum

gotestsum runs tests using go test --json, prints formatted test output, and a summary of the test run. It is designed to work well for both local development, and for automation like CI. gotest.tools/gotestsum/testjson (godoc) is a library that can be used to read test2json output.

See documentation.

Install

Download a binary from releases, or build from source with go get gotest.tools/gotestsum.

Demo

A demonstration of three --format options.

Demo
Source

Documentation

Output Format

The --format flag or GOTESTSUM_FORMAT environment variable set the format that is used to print the test names, and possibly test output, as the tests run. Most outputs use color to highlight pass, fail, or skip.

Commonly used formats (see --help for a full list):

  • dots - print a character for each test.
  • pkgname (default) - print a line for each package.
  • testname - print a line for each test and package.
  • standard-quiet - the standard go test format.
  • standard-verbose - the standard go test -v format.

Have an idea for a new format? Please share it on github!

Summary

Following the formatted output is a summary of the test run. The summary includes:

  • The test output, and elapsed time, for any test that fails or is skipped.

  • The build errors for any package that fails to build.

  • A DONE line with a count of tests run, tests skipped, tests failed, package build errors, and the elapsed time including time to build.

    DONE 101 tests[, 3 skipped][, 2 failures][, 1 error] in 0.103s
    

To hide parts of the summary use --hide-summary section.

Example: hide skipped tests in the summary

gotestsum --hide-summary=skipped

Example: hide everything except the DONE line

gotestsum --hide-summary=skipped,failed,errors,output
# or
gotestsum --hide-summary=all

Example: hide test output in the summary, only print names of failed and skipped tests and errors

gotestsum --hide-summary=output

JUnit XML output

When the --junitfile flag or GOTESTSUM_JUNITFILE environment variable are set to a file path, gotestsum will write a test report, in JUnit XML format, to the file. This file can be used to integrate with CI systems.

gotestsum --junitfile unit-tests.xml

If the package names in the testsuite.name or testcase.classname fields do not work with your CI system these values can be customized using the --junitfile-testsuite-name, or --junitfile-testcase-classname flags. These flags accept the following values:

  • short - the base name of the package (the single term specified by the package statement).
  • relative - a package path relative to the root of the repository
  • full - the full package path (default)

Note: If Go is not installed, or the go binary is not in PATH, the GOVERSION environment variable can be set to remove the "failed to lookup go version for junit xml" warning.

JSON file output

When the --jsonfile flag or GOTESTSUM_JSONFILE environment variable are set to a file path, gotestsum will write a line-delimited JSON file with all the test2json output that was written by go test --json. This file can be used to compare test runs, or find flaky tests.

gotestsum --jsonfile test-output.log

Post Run Command

The --post-run-command flag may be used to execute a command after the test run has completed. The binary will be run with the following environment variables set:

GOTESTSUM_FORMAT        # gotestsum format (ex: short)
GOTESTSUM_JSONFILE      # path to the jsonfile, empty if no file path was given
GOTESTSUM_JUNITFILE     # path to the junit.xml file, empty if no file path was given
TESTS_ERRORS            # number of errors
TESTS_FAILED            # number of failed tests
TESTS_SKIPPED           # number of skipped tests
TESTS_TOTAL             # number of tests run

To get more details about the test run, such as failure messages or the full list of failed tests, run gotestsum with either a --jsonfile or --junitfile and parse the file from the post-run-command. The gotestsum/testjson package may be used to parse the JSON file output.

Example: desktop notifications

First install the example notification command with go get gotest.tools/gotestsum/contrib/notify. The command will be downloaded to $GOPATH/bin as notify. Note that this example notify command only works on macOS with terminal-notifer installed.

gotestsum --post-run-command notify

Re-running failed tests

When the --rerun-fails flag is set, gotestsum will re-run any failed tests. The tests will be re-run until each passes once, or the number of attempts exceeds the maximum attempts. Maximum attempts defaults to 2, and can be changed with --rerun-fails=n.

To avoid re-running tests when there are real failures, the re-run will be skipped when there are too many test failures. By default this value is 10, and can be changed with --rerun-fails-max-failures=n.

Note that using --rerun-fails may require the use of other flags, depending on how you specify args to go test:

  • when used with --raw-command the re-run will pass additional arguments to the command. The first arg is a -test.run flag with a regex that matches the test to re-run, and second is the name of a go package. These additional args can be passed to go test, or a test binary.

  • when used with any go test args (anything after -- on the command line), the list of packages to test must be specified as a space separated list using the --packages arg.

    Example

    gotestsum --rerun-fails --packages="./..." -- -count=2
    
  • if any of the go test args should be passed to the test binary, instead of go test itself, the -args flag must be used to separate the two groups of arguments. -args is a special flag that is understood by go test to indicate that any following args should be passed directly to the test binary.

    Example

    gotestsum --rerun-fails --packages="./..." -- -count=2 -args -update-golden
    

Custom go test command

By default gotestsum runs tests using the command go test --json ./.... You can change the command with positional arguments after a --. You can change just the test directory value (which defaults to ./...) by setting the TEST_DIRECTORY environment variable.

You can use --debug to echo the command before it is run.

Example: set build tags

gotestsum -- -tags=integration ./...

Example: run tests in a single package

gotestsum -- ./io/http

Example: enable coverage

gotestsum -- -coverprofile=cover.out ./...

Example: run a script instead of go test

gotestsum --raw-command -- ./scripts/run_tests.sh

Note: when using --raw-command you must ensure that the stdout produced by the script only contains the test2json output. Any stderr produced by the script will be considered an error (this behaviour is necessary because package build errors are only reported by writting to stderr, not the test2json stdout). Any stderr produced by tests is not considered an error (it will be in the test2json stdout).

Example: using TEST_DIRECTORY

TEST_DIRECTORY=./io/http gotestsum

Executing a compiled test binary

gotestsum supports executing a compiled test binary (created with go test -c) by running it as a custom command.

The -json flag is handled by go test itself, it is not available when using a compiled test binary, so go tool test2json must be used to get the output that gotestsum expects.

Example: running ./binary.test

gotestsum --raw-command -- go tool test2json -t -p pkgname ./binary.test -test.v

pkgname is the name of the package being tested, it will show up in the test output. ./binary.test is the path to the compiled test binary. The -test.v must be included so that go tool test2json receives all the output.

To execute a test binary without installing Go, see running without go.

Finding and skipping slow tests

gotestsum tool slowest reads test2json output, from a file or stdin, and prints the names and elapsed time of slow tests. The tests are sorted from slowest to fastest.

gotestsum tool slowest can also rewrite the source of tests slower than the threshold, making it possible to optionally skip them.

The test2json output can be created with gotestsum --jsonfile or go test -json.

See gotestsum tool slowest --help.

Example: printing a list of tests slower than 500 milliseconds

$ gotestsum --format dots --jsonfile json.log
[.]····↷··↷·
$ gotestsum tool slowest --jsonfile json.log --threshold 500ms
gotest.tools/example TestSomething 1.34s
gotest.tools/example TestSomethingElse 810ms

Example: skipping slow tests with go test --short

Any test slower than 200 milliseconds will be modified to add:

if testing.Short() {
    t.Skip("too slow for testing.Short")
}
go test -json -short ./... | gotestsum tool slowest --skip-stmt "testing.Short" --threshold 200ms

Use git diff to see the file changes. The next time tests are run using --short all the slow tests will be skipped.

Run tests when a file is saved

When the --watch flag is set, gotestsum will watch directories using file system notifications. When a Go file in one of those directories is modified, gotestsum will run the tests for the package which contains the changed file. By default all directories under the current directory will be watched. Use the --packages flag to specify a different list.

While in watch mode, pressing some keys will perform an action:

  • r will run tests for the previous event.
  • d will run tests for the previous event using dlv test, allowing you to debug a test failure using delve A breakpoint will automatically be added at the first line of any tests which failed in the previous run. Additional breakpoints can be added with runtime.Breakpoint or by using the delve command prompt.

Note that delve must be installed in order to use debug (d).

Example: run tests for a package when any file in that package is saved

gotestsum --watch --format testname

Development

Godoc CircleCI Go Reportcard

Pull requests and bug reports are welcome! Please open an issue first for any big changes.

Thanks

This package is heavily influenced by the pytest test runner for python.

Comments
  • Passing tests reported as failures in summary

    Passing tests reported as failures in summary

    We are using gotestsum to run our testing for Knative, and digging into some recent "failures", I can't really make sense of why gotestsum is reporting them as failed.

    The failure log I have been digging through can be downloaded from here, the file you want is "build-log.txt" (it is large).

    In this test run I see the following summary:

    === Skipped
    === SKIP: test/e2e TestProbeWhitelist (0.00s)
        probe_whitelist_test.go:38: RequestAuthentication dropped the regex support. We need to rewrite this test. See https://github.com/istio/istio/issues/16585
    
    
    === Failed
    === FAIL: test/conformance/api/v1alpha1 TestContainerExitingMsg/http (unknown)
    
    === FAIL: test/conformance/api/v1alpha1 TestContainerExitingMsg/tcp (unknown)
    

    However, if I scan back through the test logs, I see that all of the flavors of TestContainerExitingMsg have passed, here's one cluster:

    --- PASS: TestContainerExitingMsg (0.00s)
        --- PASS: TestContainerExitingMsg/http (40.29s)
        --- PASS: TestContainerExitingMsg/tcp (172.49s)
    

    I've been trying to make sense of the pieces of information here, but basically all I've managed so far is that (unknown) seems to tell us that the elapsed time is unknown, which seems to point to a TestCase terminated by the end() method here.

    Any pointers would be appreciated. 🙏

  • Add tool ci-matrix

    Add tool ci-matrix

    This PR adds a new subcommand gotestsum tool ci-matrix for use with github actions. The subcommand:

    • reads test2json files saved in a github actions cache from previous test runs
    • uses the test2json data to calculate the 85th percentile runtime of each package
    • uses the package runtime to bucket all the packages into n buckets

    A github actions workflow can then use those buckets in a matrix, so that packages are split between the buckets based on their runtime. This should result in optimal splitting of packages to minimize overall CI runtime.

    Using the subcommand requires integration with github actions, so I'll need to document exact how to wire it all together. Once it's working well, maybe it would make sense to publish an action to handle parts of it.

  • support goarch mips64le architecture.

    support goarch mips64le architecture.

    hello,I am going to submit mips64le architecture,The main reasons for adding support for mips64le architecture are: 1、The mips64le architecture is also the mainstream cpu architecture (amd64, arm64), which is used by many users; 2、Golang supports cross compilation, and mips64le is officially supported. Therefore, I hope that the release package also supports the mips64le architecture, which is convenient for more users. Signed-off-by: houfangdong [email protected]

  • Add attribute `file` to JUnit XML report to accommodate CircleCI algorithm for splitting tests

    Add attribute `file` to JUnit XML report to accommodate CircleCI algorithm for splitting tests

    Hi,

    CircleCI allows to split tests by timings using a JUnit XML report. Unfortunately it requires testcase to have an file attribute in order to properly work as detailed in this CircleCI forum question. I have confirmed it by manipulating the JUnit XML report output of gotestsum. I understand that file is not part of the JUnit XSD but it would really help if it could be added to the output if the user wanted it. What are your thoughts?

  • Execution time is 0.00s when using gotestsum with binary.

    Execution time is 0.00s when using gotestsum with binary.

    I'am trying to use gotestsum in following way: gotestsum -f short-verbose --raw-command -- go tool test2json -p test ./test/run-tests -test.v

    Test are running without problem but time execution of any test is 0.00s. F.e.: PASS test.TestSomeFoo (0.00s)

    Overall test execution time is reported properly. Am I missing something?

  • dots formatting broken in v0.4.1

    dots formatting broken in v0.4.1

    Upgrading to v0.4.1 has broken dots formatting. It's now printing out a huge amount of output like this:

        🖴  internal/pkg/jwt ·······
        🖴  internal/pkg/random ·
        🖴  internal/pkg/httputil ···············
        🖴  internal/pkg/retry ···
        🖴  internal/migrator ··
        🖴  internal/api/audit ·········
        🖴  internal/blah··········································
    

    command used:

    gotestsum --format dots ./...
    

    platform: OSX 10.15.1

    Cheers

  • [go1.13.x] Error handling TestMain logic

    [go1.13.x] Error handling TestMain logic

    If TestMain exits 0 (or just returns) and no tests are executed, the package will show as failed. See: https://github.com/DataDog/datadog-agent/pull/5294

    func TestMain(m *testing.M) {
    	if _, ok := os.LookupEnv("INTEGRATION"); !ok {
    		log.Println("--- SKIP: to run tests in this package, set the INTEGRATION environment variable")
    		os.Exit(0)
    	}
    	os.Exit(m.Run())
    }
    
    === Failed
    === FAIL: pkg/trace/test/testsuite  (0.00s)
    2020/04/15 15:35:55 --- SKIP: to run tests in this package, set the INTEGRATION environment variable
    ok  	github.com/DataDog/datadog-agent/pkg/trace/test/testsuite	1.015s
    
  • Module go download logs appearing as Errors (go v1.11)

    Module go download logs appearing as Errors (go v1.11)

    Hey, I'm using the following command to run tests:

    gotestsum -- ./api/...
    

    inside a golang:1.11.1-stretch docker container, and also locally (mac), and getting the following output

    ∅ bitbucket.com/satalia/engine/api/healthsvc
    ∅ bitbucket.com/satalia/engine/api/healthsvc/endpoints
    ✓ bitbucket.com/satalia/engine/api/healthsvc/service (5ms)
    ∅ bitbucket.com/satalia/engine/api/healthsvc/transport
    
    === Errors
    go: downloading github.com/go-kit/kit v0.7.0
    go: downloading github.com/golang/protobuf v1.2.0
    go: downloading google.golang.org/grpc v1.15.0
    go: downloading golang.org/x/net v0.0.0-20181011144130-49bb7cea24b1
    go: downloading google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8
    go: downloading golang.org/x/text v0.3.0
    go: downloading github.com/go-logfmt/logfmt v0.3.0
    go: downloading github.com/go-stack/stack v1.8.0
    
    DONE 1 tests, 8 errors in 29.465s
    

    whereas if I run using go test ./api/... I get none of the go downloading errors.

    We are using go modules to manage dependencies (a file called go.mod) contains all the required packages, could this be an issue?

  • Duplicate test names in JUnit XML

    Duplicate test names in JUnit XML

    I used this invocation of gotestsum:

    gotestsum --format standard-verbose --junitfile report.xml --jsonfile report.json -- -test.run TestMySuite
    

    The output XML looks like:

    <?xml version="1.0" encoding="UTF-8"?>
    <testsuites>
            <testsuite tests="11" failures="0" time="5806.690s" name="MySuite">
                    <properties>
                            <property name="go.version" value="go1.10.3"></property>
                    </properties>
                    <testcase classname="job" name="TestOne" time="75.350s"></testcase>
                    <testcase classname="job" name="TestOne" time="76.680s"></testcase>
                    <testcase classname="job" name="TestTwo" time="86.110s"></testcase>
                    <testcase classname="job" name="TestTwo" time="86.990s"></testcase>
                    <testcase classname="job" name="TestThree" time="188.780s"></testcase>
                    <testcase classname="job" name="TestThree" time="189.740s"></testcase>
                    <testcase classname="job" name="TestMySuite" time="11.570s"></testcase>
                    <testcase classname="job" name="TestMySuite" time="1937.300s"></testcase>
            </testsuite>
    </testsuites>
    

    The JSON is pretty big, but if I cut it down, it looks something like:

    {"Time":"2018-08-02T08:18:13.739327862Z","Action":"run","Package":"MySuite","Test":"TestOne"}
    {"Time":"2018-08-02T08:19:30.419938008Z","Action":"pass","Package":"MySuite","Test":"TestOne","Elapsed":75.35}
    {"Time":"2018-08-02T08:19:30.419959464Z","Action":"output","Package":"MySuite","Test":"TestOne","Output":"--- PASS: TestOne (76.68s)\n"}
    {"Time":"2018-08-02T08:19:30.419971053Z","Action":"pass","Package":"MySuite","Test":"TestOne","Elapsed":76.68}
    {"Time":"2018-08-02T08:19:30.419976409Z","Action":"run","Package":"MySuite","Test":"TestTwo"}
    {"Time":"2018-08-02T08:20:57.405652904Z","Action":"pass","Package":"MySuite","Test":"TestTwo","Elapsed":86.11}
    {"Time":"2018-08-02T08:20:57.405663056Z","Action":"output","Package":"MySuite","Test":"TestTwo","Output":"--- PASS: TestTwo (86.99s)\n"}
    {"Time":"2018-08-02T08:20:57.405671178Z","Action":"pass","Package":"MySuite","Test":"TestTwo","Elapsed":86.99}
    {"Time":"2018-08-02T08:20:57.405676436Z","Action":"run","Package":"MySuite","Test":"TestThree"}
    {"Time":"2018-08-02T08:24:07.146323211Z","Action":"pass","Package":"MySuite","Test":"TestThree","Elapsed":188.78}
    {"Time":"2018-08-02T08:24:07.146333316Z","Action":"output","Package":"MySuite","Test":"TestThree","Output":"--- PASS: TestThree (189.74s)\n"}
    {"Time":"2018-08-02T08:24:07.146340153Z","Action":"pass","Package":"MySuite","Test":"TestThree","Elapsed":189.74}
    
    {"Time":"2018-08-02T08:50:27.611048188Z","Action":"pass","Package":"MySuite","Test":"MySuite","Elapsed":11.57}
    {"Time":"2018-08-02T08:50:27.611767989Z","Action":"pass","Package":"MySuite","Test":"MySuite","Elapsed":1937.3}
    {"Time":"2018-08-02T08:50:27.611054838Z","Action":"output","Package":"MySuite","Test":"MySuite","Output":"--- PASS: MySuite (1937.30s)\n"}
    {"Time":"2018-08-02T08:50:27.61302664Z","Action":"pass","Package":"MySuite","Elapsed":1937.313}
    

    Any idea why lines with "Action":"pass" appear twice for the same test?

    I am using go 1.10.3

  • Hide empty packages in compact formats

    Hide empty packages in compact formats

    Related to #228 Closes #282

    Add a new --format-hide-empty-pkg flag to hide empty package in compact formats. The testname format should be verbose enough that a few extra EMPTY lines for packages is not a big deal, and the standard formats should emulate the go test output as close as possible, so we don't want it to apply to those.

    I tested both formats manually, and added golden tests for pkgname.

  • --rerun-fails do not execute remaining tests after rerun test passes

    --rerun-fails do not execute remaining tests after rerun test passes

    Hi,

    Thank you for your contribution to the gotestsum package. It is extremely useful.

    We started using the --rerun-fails flag to rerun potentially unstable test suites to prevent false failures in our CI/CD pipeline.

    However, we identified 2 issues with this:

    1. Not all tests are getting executed when the previously failing test passes. For example, if the test suite has 30 tests, and the 5th test fails, then gotestsum retries the 5th test, and if it passes, then test execution stops there.
    2. --format short-verbose does not take effect for the failed tests. We get the standard-verbose output for the failed test case. As a result console log shows that the test suite has failed, but in reality, the test was passed in the retry.

    Below is our usage scenario. Each service has n number of tests (Service-1 has 10 tests, Service-2 has 30 tests, and Service-3 has 4 tests).

    Here is an example of gotestsum usage in Makefile BEFORE --rerun-fails

    GO_TEST = gotestsum --format short-verbose -- -count=1 -failfast
    
    test_all: service1 service2 service3
    
    service1:
       $(GO_TEST) ./services/service1/scenario
    service2:
       $(GO_TEST) ./services/service2/scenario
    service3:
       $(GO_TEST) ./services/service3/scenario
    
    

    Here is an example of gotestsum usage in Makefile AFTER --rerun-fails

    define go_test
       gotestsum --rerun-fails --packages="$(1)" --format short-verbose -- -count=1 -failfast
    endef
    
    test_all: service1 service2 service3
    
    service1:
       $(call go_test,./services/service1/scenario)
    service2:
       $(call go_test,./services/service2/scenario)
    service3:
       $(call go_test,./services/service3/scenario)
    
    

    We want to achieve the following outcome with respect to our usecase above:

    1. If any test case in service-2 fails after rerun, whole test execution should stop and service-3 tests should not be executed. (i.e. -failfast should take effect). It was working before we started using --rerun-fails.
    2. If the 5th test case in service-2 passes, after rerun from failed 1st time (i.e. failed first time, but passes second time), then the rest of the 25 tests should continue execution. Currently this is a problem with the use of --rerun-fails
    3. Log output should not be printed for the test case that passed in the re-run.

    Appreciate your help 🙏

  • Add flag to always return success status even when there are failed tests

    Add flag to always return success status even when there are failed tests

    When running tests as a part of the CI process, it's critical that the test step will always return a success status so that the process can continue and just present tests results report. At the moment gotestsum returns non-zero status and causes our build to fail. On commandline it looks like this:

    ➜  blink git:(xxx) ✗ gotestsum -- ./pkg/errors/...
    ✖  pkg/errors (445ms)
    
    === Failed
    === FAIL: pkg/errors TestIsDuplicateKeyError/not_duplicate_key_error_1 (0.00s)
        errors_test.go:127: IsSQLError() = false, want true
        --- FAIL: TestIsDuplicateKeyError/not_duplicate_key_error_1 (0.00s)
    
    === FAIL: pkg/errors TestIsDuplicateKeyError (0.00s)
    
    DONE 18 tests, 2 failures in 1.079s
    ➜  blink git:(xxx) ✗ echo $?
    1
    

    At the moment I have no choice but add || true at the end of the command which is obviously bad practice - if there is a real failure (not test failure), I won't be able to catch it. Please add a flag that will cause gotestsum to ignore test fails, I didn't find one in the documentation..

  • Accept additional json TestEvents from a side channel

    Accept additional json TestEvents from a side channel

    Background

    This proposal aims to address two problems:

    1. log output from background goroutines can not be correctly attributed to the right test when t.Parallel is used. I no longer use t.Parallel so it's hard for me to verify this is still a problem (https://github.com/golang/go/commit/1c72ee7f13831b215b8744f6b35bc4fd53aba5e2 may have fixed this, to be released in Go1.20). There are test suites out there that use t.Parallel extensively, and when I worked on them, this was a big problem. I believe the Go team has said this is working as intended and there is no way to fix this in the stdlib (TODO: find the relevant issue on the Go issue tracker). That's only true because the test binary doesn't directly output JSON, but it seems like that is not going to change.

    2. tracking the timing of TestMain (see https://github.com/gotestyourself/gotestsum/issues/220#issuecomment-1352688384)

    Proposal

    Note this is a big feature, so it should not be implemented unless there is really no way to avoid it.

    A test suite can't send TestEvent json to gotestsum directly, because any output from the test suite is wrapped by test2json. For gotestsum to receive custom events there needs to be a side channel, a different output stream (not stdout or stderr).

    I'm not sure if gotestsum could use ExtraFiles on https://pkg.go.dev/os/exec#Cmd when running the tests. I suspect because go test is between gotestsum and the test binary that the file descriptors won't be passed along. So most likely we'd have to use a file or pipe as a side channel for these events. A regular file is probably easier and more portable.

    Essentially gotestsum would have a flag to specify a filename for side channel events, and would run go test with an extra flag to specify that filename:

    gotestsum --extra-events-file=/tmp/testevents.log ./...
    

    would run

    go test ./... --extra-events-file=/tmp/testevents.log
    

    The test suite would have to get the value from --extra-events-file, open that file, and write the newline delimited JSON events to it, as described here: https://pkg.go.dev/cmd/test2json#hdr-Output_Format

    gotestsum would open that file as another source of events in https://github.com/gotestyourself/gotestsum/blob/10c2a47f9b99c8b49cbea60d12a6c76af0678d50/testjson/execution.go#L648-L654 and merge those into the Execution.

    gotestsum not only needs the new flag, but testjson.Execution would also need to use locks to synchronize writes (or maybe build two separate Execution and merge them afterward?). And the test suite would also have to do some work to write those events to a file.

    Maybe there's an easier way to create a side channel of events?

  • Issue new version release

    Issue new version release

    Hi! Maybe it’s time to issue a new release tag? It seems a bunch of changes have arrived since the latest release: https://github.com/gotestyourself/gotestsum/compare/v1.8.2...HEAD

    @aaronlehmann @dnephin @gotestyourself-bot @vdemeester

  • Add option to use high-visibility icons

    Add option to use high-visibility icons

    The default icons used in the pkgname output formats (∅,✓,𐄂) can be hard to spot when scrolling through test output.

    This PR adds the ability to pick a higher-visiblity/contrast set of icons (🔳,✅,❌) by adding a -hivis suffix to either of the 'pkgname' style formats.

  • Panic stack trace from `go test -timeout=X` is ignored when timeout happens right after a test passes

    Panic stack trace from `go test -timeout=X` is ignored when timeout happens right after a test passes

    This seems like a bug in go test -json, so I've opened the following issue over there: https://github.com/golang/go/issues/57305

    However, this is still relevant for gotestsum since previous Go versions are affected.

    The following visual should give an idea of what's wrong:

    ❯ gotestsum --jsonfile=trace-missing.json -- . -timeout=1s
    ✖  test (1.012s)
    
    === Failed
    === FAIL: test  (0.00s)
    FAIL	github.com/mafredri/test	1.012s
    
    DONE 1 tests, 1 failure in 1.323s
    

    Something went wrong, but we simply don't know what. With parallel tests this is even more confusing:

    ❯ gotestsum --jsonfile=parallel-trace-missing.json -- . -timeout=1s
    ✖  test (1.008s)
    
    === Failed
    === FAIL: test TestWorld (unknown)
    
    DONE 2 tests, 1 failure in 1.326s
    

    The test fails with (unknown), but no explanation.

    When this edge case isn't hit we see the proper output:

    ❯ gotestsum --jsonfile=trace-valid.json -- . -timeout=1s
    ✖  test (1.01s)
    
    === Failed
    === FAIL: test TestWorld (unknown)
    panic: test timed out after 1s
    running tests:
    	TestWorld (0s)
    
    goroutine 5 [running]:
    [...]
    
    DONE 2 tests, 1 failure in 1.317s
    

    The json payloads are available at https://github.com/golang/go/issues/57305, but for context, this is the significant portion:

    {"Time":"2022-12-14T09:49:02.56982657Z","Action":"output","Package":"github.com/mafredri/test","Test":"TestHello","Output":"--- PASS: TestHello (1.00s)\n"}
    {"Time":"2022-12-14T09:49:02.572963923Z","Action":"output","Package":"github.com/mafredri/test","Test":"TestHello","Output":"panic: test timed out after 1s\n"}
    {"Time":"2022-12-14T09:49:02.572982687Z","Action":"output","Package":"github.com/mafredri/test","Test":"TestHello","Output":"running tests:\n"}
    {"Time":"2022-12-14T09:49:02.572992095Z","Action":"output","Package":"github.com/mafredri/test","Test":"TestHello","Output":"\tTestHello (1s)\n"}
    

    What's going on here is that the test outputs PASS, relied on by gotestsum to ignore future output from that test. However, go test considers that test to be running still for a brief moment and attributes the panic stack trace to it.

  • Allow re-running `panic`'ed tests through a runtime configuration flag

    Allow re-running `panic`'ed tests through a runtime configuration flag

    I'm using gotestsum to test a terraform provider that frequently has a very flaky acceptance test build (context in https://github.com/terra-farm/terraform-provider-xenorchestra/issues/188). The flakiness is due to tests becoming hung and eventually reaching go test's timeout value. Re-running these tests results in a successful build, but requires a significant amount of manual work to identify what should be rerun.

    I've tested a fork of gotestsum that allows rerunning panic'ed tests (https://github.com/ddelnano/gotestsum/commit/8c70ff05e83bc93e5e042151a4d25343f1dfe4f8) and it appears to work the way I intend it to -- tests that timeout are re-run and result in a passing test build. I created a dummy test package to test this. The test package includes 3 tests and will allow one of the tests to proceed with execution every 10 seconds. If go test is given a timeout greater than 30s all three tests pass. If the timeout is between 20s and 30s, then 1 of the tests will timeout and panic.

    I used this package to test the behavior of my fork and verified that the result is expected.
    # Verify that gotestsum re-runs the panic'ed tests and reports a successful build (exit code 0)
    ddelnano@ddelnano-desktop:~/go/src/github.com/ddelnano/terraform-provider-xenorchestra$ ~/code/gotestsum/gotestsum --rerun-fails=2  --packages='github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel'  -- -v -count=1 -timeout=21s
    ✖  cmd/testing/parallel (21.016s)
    
    DONE 3 tests, 1 failure in 21.846s
    
    ✓  cmd/testing/parallel (10.031s)
    
    === Failed
    === FAIL: cmd/testing/parallel TestParallelHangging (unknown)
    
    DONE 2 runs, 4 tests, 1 failure in 32.883s
    
    ddelnano@ddelnano-desktop:~/go/src/github.com/ddelnano/terraform-provider-xenorchestra$ echo $?
    0
    
    
    # Force test to always fail and verify that gotestsum reports a failure
    ddelnano@ddelnano-desktop:~/go/src/github.com/ddelnano/terraform-provider-xenorchestra$ git diff
    diff --git a/cmd/testing/parallel/main_test.go b/cmd/testing/parallel/main_test.go
    index b47d341..1c9cfc9 100644
    --- a/cmd/testing/parallel/main_test.go
    +++ b/cmd/testing/parallel/main_test.go
    @@ -38,4 +38,5 @@ func TestParallelHangging(t *testing.T) {
     func TestParallelSlightHangging(t *testing.T) {
            t.Parallel()
            waitForChannel(c)
    +       t.Errorf("Failed")
     }
    
    
    ddelnano@ddelnano-desktop:~/go/src/github.com/ddelnano/terraform-provider-xenorchestra$ ~/code/gotestsum/gotestsum --rerun-fails=2  --packages='github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel'  -- -v -count=1 -timeout=21s || echo $?
    ✖  cmd/testing/parallel (21.008s)
    
    DONE 3 tests, 2 failures in 21.384s
    
    ✖  cmd/testing/parallel (10.014s)
    ✓  cmd/testing/parallel (10.012s)
    
    DONE 2 runs, 5 tests, 3 failures in 42.515s
    
    ✖  cmd/testing/parallel (10.011s)
    
    === Failed
    === FAIL: cmd/testing/parallel TestParallelSlightHangging (20.01s)
        main_test.go:41: Failed
    panic: test timed out after 21s
    
    goroutine 19 [running]:
    testing.(*M).startAlarm.func1()
            /usr/lib/go/src/testing/testing.go:2036 +0x8e
    created by time.goFunc
            /usr/lib/go/src/time/sleep.go:176 +0x32
    
    goroutine 1 [chan receive]:
    testing.tRunner.func1()
            /usr/lib/go/src/testing/testing.go:1412 +0x4a5
    testing.tRunner(0xc000116680, 0xc000106c78)
            /usr/lib/go/src/testing/testing.go:1452 +0x144
    testing.runTests(0xc0001300a0?, {0x605400, 0x3, 0x3}, {0x4a6be9?, 0x7f497f542fff?, 0x609ca0?})
            /usr/lib/go/src/testing/testing.go:1844 +0x456
    testing.(*M).Run(0xc0001300a0)
            /usr/lib/go/src/testing/testing.go:1726 +0x5d9
    github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel.TestMain(0x7f49a6674a68?)
            /home/ddelnano/go/src/github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel/main_test.go:21 +0x5e
    main.main()
            _testmain.go:53 +0x1d3
    
    goroutine 34 [sleep]:
    time.Sleep(0x2540be400)
            /usr/lib/go/src/runtime/time.go:195 +0x135
    github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel.TestMain.func1()
            /home/ddelnano/go/src/github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel/main_test.go:16 +0x2c
    created by github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel.TestMain
            /home/ddelnano/go/src/github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel/main_test.go:14 +0x54
    
    goroutine 36 [chan receive]:
    github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel.waitForChannel(...)
            /home/ddelnano/go/src/github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel/main_test.go:25
    github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel.TestParallelHangging(0x0?)
            /home/ddelnano/go/src/github.com/ddelnano/terraform-provider-xenorchestra/cmd/testing/parallel/main_test.go:35 +0x27
    testing.tRunner(0xc000116b60, 0x531050)
            /usr/lib/go/src/testing/testing.go:1446 +0x10b
    created by testing.(*T).Run
            /usr/lib/go/src/testing/testing.go:1493 +0x35f
    
    === FAIL: cmd/testing/parallel TestParallelHangging (unknown)
    
    === FAIL: cmd/testing/parallel TestParallelSlightHangging (re-run 1) (10.01s)
        main_test.go:41: Failed
    
    === FAIL: cmd/testing/parallel TestParallelSlightHangging (re-run 2) (10.01s)
        main_test.go:41: Failed
    
    DONE 3 runs, 6 tests, 4 failures in 53.302s
    1
    

    While working on this, I realized that there is a history on handling this situation (https://github.com/gotestyourself/gotestsum/pull/192 and https://github.com/golang/go/issues/45508), so I know it was intentional that these tests aren't rerun but I'd like to make it possible to enable this behavior behind a new cli flag (--rerun-fails-rerun-panics or something similar).

    I'm opening this issue to discuss if a PR that implements this would be accepted. If that is the case, I'm happy to take on this work myself.

Decode / encode XML to/from map[string]interface{} (or JSON); extract values with dot-notation paths and wildcards. Replaces x2j and j2x packages.

mxj - to/from maps, XML and JSON Decode/encode XML to/from map[string]interface{} (or JSON) values, and extract/modify values from maps by key or key-

Dec 29, 2022
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.
omniparser: a native Golang ETL streaming parser and transform library for CSV, JSON, XML, EDI, text, etc.

omniparser Omniparser is a native Golang ETL parser that ingests input data of various formats (CSV, txt, fixed length/width, XML, EDI/X12/EDIFACT, JS

Jan 4, 2023
Convert xml and json to go struct

xj2go The goal is to convert xml or json file to go struct file. Usage Download and install it: $ go get -u -v github.com/wk30/xj2go/cmd/... $ xj [-t

Oct 23, 2022
parse and generate XML easily in go

etree The etree package is a lightweight, pure go package that expresses XML in the form of an element tree. Its design was inspired by the Python Ele

Dec 19, 2022
Go XML sitemap and sitemap index generator

Install go get github.com/turk/go-sitemap Example for sitemapindex func () main(c *gin.Context) { s := sitemap.NewSitemapIndex(c.Writer, true)

Jun 29, 2022
This package provides Go (golang) types and helper functions to do some basic but useful things with mxGraph diagrams in XML, which is most famously used by app.diagrams.net, the new name of draw.io.

Go Draw - Golang MX This package provides types and helper functions to do some basic but useful things with mxGraph diagrams in XML, which is most fa

Aug 30, 2022
Quick and simple parser for PFSense XML configuration files, good for auditing firewall rules

pfcfg-parser version 0.0.1 : 13 January 2022 A quick and simple parser for PFSense XML configuration files to generate a plain text file of the main c

Jan 13, 2022
Sqly - An easy-to-use extension for sqlx, base on xml files and named query/exec

sqly An easy-to-use extension for sqlx ,base on xml files and named query/exec t

Jun 12, 2022
xmlquery is Golang XPath package for XML query.

xmlquery Overview xmlquery is an XPath query package for XML documents, allowing you to extract data or evaluate from XML documents with an XPath expr

Jan 1, 2023
Extraction politique de conformité : xlsx (fichier de suivi) -> xml (format AlgoSec)

go_policyExtractor Extraction politique de conformité : xlsx (fichier de suivi) -> xml (format AlgoSec). Le programme suivant se base sur les intitulé

Nov 4, 2021
axmlfmt is an opinionated formatter for Android XML resources

axmlfmt axmlfmt is an opinionated formatter for Android XML resources. It takes XML that looks like <?xml version="1.0" encoding="utf-8"?> <LinearLayo

May 14, 2022
Freestyle xml parser with golang

fxml - FreeStyle XML Parser This package provides a simple parser which reads a XML document and output a tree structure, which does not need a pre-de

Jul 1, 2022
🧑‍💻 Go XML generator without Structs™

exml ??‍?? Go XML generator without Structs™ Package exml allows XML documents to be generated without the usage of structs or maps. It is not intende

Nov 15, 2022
wikipedia-jsonl is a CLI that converts Wikipedia dump XML to JSON Lines format.

wikipedia-jsonl wikipedia-jsonl is a CLI that converts Wikipedia dump XML to JSON Lines format. How to use At first, download the XML dump from Wikime

Dec 26, 2022
:sushi: emoji terminal output for golang

Emoji Emoji is a simple golang package. Get it: go get github.com/kyokomi/emoji/v2 Import it: import ( "github.com/kyokomi/emoji/v2" ) Usage packag

Jan 8, 2023
Parse data and test fixtures from markdown files, and patch them programmatically, too.

go-testmark Do you need test fixtures and example data for your project, in a language agnostic way? Do you want it to be easy to combine with documen

Oct 31, 2022
Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler
Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

Pagser Pagser inspired by page parser。 Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and str

Dec 13, 2022
[Go] Package of validators and sanitizers for strings, numerics, slices and structs

govalidator A package of validators and sanitizers for strings, structs and collections. Based on validator.js. Installation Make sure that Go is inst

Dec 28, 2022
Take screenshots of websites and create PDF from HTML pages using chromium and docker

gochro is a small docker image with chromium installed and a golang based webserver to interact wit it. It can be used to take screenshots of w

Nov 23, 2022