Hey Mat, long time. Ran into a curious issue worth filing.
Summary
Running go test -json ...
and using this package reports incorrect test names in JSON events. This could very well be a bug in Go (test2json), but figured I'd start here because if I remove is
and use regular std. lib primitives it reports the correct thing.
Reproducible example
I noticed this happening in https://github.com/pressly/goose and put together a PR to showcase the problem.
In this commit I purposefully changed a test to trigger a failure. Note the name of the test is TestNotAllowMissing
.
https://github.com/pressly/goose/pull/353/commits/9b7c7321bdbabb8307e71a40e0601d86b222d6ce#diff-080f0d7ba408eaf6181e1f61388995ebcd57fdd1468ffafffbf789176531418fR30
When I run go test -v -json -count=1 ./tests/e2e -dialect=postgres
I happily get JSON output. But, the test name in the JSON event doesn't line up with the Output.
I was expecting Output that is
prints to be associated with test named TestNotAllowMissing
, but instead it was associated with another test named TestMigrateFull
. (the name is non-deterministic, it changes between runs).
The raw output can be found here. But I've copied and trimmed the relevant bits:
{"Test":"TestMigrateFull","Output":"\t\u001b[90mallow_missing_test.go:30: \u001b[39m7 != 8\n"}
{"Test":"TestNotAllowMissing","Output":"--- FAIL: TestNotAllowMissing (3.73s)\n"}
The first JSON event refers to allow_missing_test.go:30
, in the code base the call site on line 30 is located within the test named: TestNotAllowMissing
, but the test name is TestMigrateFull
.
The second JSON event correctly outputs the failure under the test named TestNotAllowMissing
.
This could very well be a bug in how go test prints JSON events, but I suspect it may be something within this library. I say this because if I remove is
from that specific test (TestNotAllowMissing
) I see the expected output associated with the correct Test name:
Raw output can be found here
{"Test":"TestNotAllowMissing","Output":" allow_missing_test.go:35: got 7: want: 8\n"}
{"Test":"TestNotAllowMissing","Output":"--- FAIL: TestNotAllowMissing (2.81s)\n"}