This PR adds 2 new flags:

-f `regexp`, hides trace headers that matches the regexp.
-m `regexp`, shows only the trace headers that matches the regexp.

Example:

➤ go run main.go -m 'PutTask|RemoveTask' ../../xxx/xxx/crash.log
1: semacquire [26 minutes] [Created by reportQueue.PutTask @ q.go:104]
    sync          sema.go:297                 runtime_notifyListWait(#343, #735)
    sync          cond.go:57                  (*Cond).Wait(#342)
    io            pipe.go:47                  (*pipe).read(#341, #348, 0x8000, 0x8000, 0, 0, 0)
    io            pipe.go:130                 (*PipeReader).Read(#588, #348, 0x8000, 0x8000, 0x8000, 0x8000, #13)
    io            io.go:390                   copyBuffer(#1482, #1125, #7, #588, #348, 0x8000, 0x8000, #1482, 0x4a80ac, 0x25)
    io            io.go:360                   Copy(#1482, #1125, #7, #588, 0xbdc020, 0, #1482)
    iodb          bucket.go:206               (*bucket).PutTimed.func1(#1482, #1125, #1125, #1482)
    iodb          bucket.go:167               (*bucket).PutTimedFunc(#62, #298, 0x20, #1124, #1383, 0, 0, 0, 0, 0)
    iodb          bucket.go:207               (*bucket).PutTimed(#62, #298, 0x20, #7, #588, #1383, 0, 0, 0, 0xa15711, ...)
    reportQueue   q.go:99                     PutTask.func1(#437, #1383, #395, 0x4, #1193, #303, #44)

Fixes #61.

In Go 1.17, the format of function arguments printed in stack traces was improved. This is mentioned in the release notes: https://tip.golang.org/doc/go1.17#compiler

The format of stack traces from the runtime (printed when an uncaught panic occurs, or when runtime.Stack is called) is improved. Previously, the function arguments were printed as hexadecimal words based on the memory layout. Now each argument in the source code is printed separately, separated by commas. Aggregate-typed (struct, array, string, slice, interface, and complex) arguments are delimited by curly braces. A caveat is that the value of an argument that only lives in a register and is not stored to memory may be inaccurate. Function return values (which were usually inaccurate) are no longer printed.

This runtime change was primarily performed in https://go-review.googlesource.com/c/go/+/304470. The description on that CL presents the change in greater detail.

This commit adds support to panicparse for parsing these new aggregate-typed arguments. It does so while retaining the type structure from the stack traces, which allows for the structure to be faithfully recreated and for aggregate-type information to be accurately augmented, when available. The aggregate-type structure is stored in a tree of mutually recursive Args and Arg structs.

The commit also adds support for handling the new "offset too large" operator that was added in the same runtime CL. This operator comes in the form of an "_" scalar argument in cases where the argument's frame offset was too large (>= 0xf0), preventing the argument from being printed in the stack trace.

This change raises questions about backwards compatability in a few different areas.

• this change does not break backwards compatability with pre-1.17 stack traces. However, due to the nature of how most of the unit tests are set up (i.e. shelling out to local go binary), we are losing some test coverage of pre-1.17 stack trace parsing. Is that ok?
• similarly, this change breaks the library's tests for pre-1.17 installations of Go. As a result, it bumps the Go version run in CI. Is that ok?
• this change will not cause users of the github.com/maruel/panicparse/v2/stack library to fail to compile. However, due to the additions to the Arg struct, users of the library will fail to observe all arguments when parsing post-1.17 stack traces (but not pre-1.17 stack traces) unless they are aware of the new IsAggregate field. Is that ok?

Hey, i'm building some golang app supervising software and found your stacktracing project interesting. Is it possible to get rid of "internal" package (e.g. rename it to "reallyinternal") so I'll be able to import it and use its functions?

Repro:

(Used to point to go1.17beta1, go1.17rc1, updated to go1.17)

go get golang.org/dl/go1.17
go1.17 download
go1.17 test ./stack

TestAugment, TestPanic and TestPanicWeb fail.

Manual compile:

git clone https://go.googlesource.com/go golang
cd golang/src
./make.bash
export PATH=$(pwd)/../bin:$PATH
cd /path/to/panicparse
go test ./stack

I tried to set this up with my company's Gin application and it...just sort of failed. All debug output is gone and the only message resulting from panic output is a bufio.Scanner: token too long error. For context, here's (a shortened by ~15 frames and obfuscated version of) what normally goes to the console:

goroutine 55 [running]:
main.recovery(0xc8201f6310)
    /appdir/middleware.go:40 +0x165
main.(*DB).Update(0xc8202d1860, 0x78e0c0, 0xc82006cd00, 0xc820369440, 0xc8201e5680)
    /appdir/db.go:150 +0x1a19
main.UpdateAccountHandler(0xc8201d4430)
    /appdir/account_handlers.go:134 +0x326
main.InjectContext.func1(0xc8201664d0)
    /appdir/context.go:52 +0x61
created by net/http.(*Server).Serve
    /appdir/server.go:1910 +0x3f6
  type=stack-trace

Admittedly we've got our own output systems in place instead of just a standard panic call, but the output format doesn't seem different enough to cause a major issue...thoughts?

This was puzzling to me when trying it out - if you don't supply a filename it just hangs and can't be interrupted. The behaviour was added in a0eb0cee87d7cad0b22421baf899a09380d36fce, but I don't follow the explanation. I can understand ignoring Ctrl-\, which was added later.

this addresses a common need for me. I often want to restrict output to a certain package or set of files because i know that's where the stuff is happening that I'm looking for.

here's some ideas to make this feature more advanced:

• both conditions are currently OR'ed. may want to change to AND, or leave it up to user. we may also want to support multiple conditions. something like the below perhaps: -filter "path=foo && (package == bar || package == baz)"
• instead of substring/equality, may want to support regex, especially for path

... I haven't personally had this need though, so I would keep it simple until it's clear we want to support such fancier things.

Right now the output looks a bit like this running it through my test suite:

make ...
                       some other command
                                        another command

This fixes that by using an io.TeeReader. It does however leave the panic on the screen before parsing it, I think this is kind of a feature depending on how you look at it. The output otherwise is unaffected (save the newlines).

Hi, thanks for the great tool!

I took a large (20k lines) goroutine dump from a server and I used panicparse to analyze it. Unfortunately panicparse still emits many thousands of lines because lots of goroutines that are the same for my purposes cannot be deduplicated.

I dug into it more and I found two specific issues:

1. Panicparse compares argument values and if it doesn't think they're pointers, it wants the values to be the same. For my purposes, I really don't care about function args; I want two goroutines to be deduped if they have the same stack trace.

   In case it helps, here are two lines from two different goroutines that caused them to not be deduped (this is deep in net's inner workings, far away from my own code):

   io.copyBuffer(0x7fa5add90238, 0xc820106aa0, 0x7fa5addd01d8, 0xc820106a60, 0xc820166000, 0x8000, 0x8000, 0x38bd, 0x0, 0x0)
   io.copyBuffer(0x7fa5add90238, 0xc820028220, 0x7fa5addd01d8, 0xc820152148, 0xc820a80000, 0x8000, 0x8000, 0x98d, 0x0, 0x0)
   

   My quick hack was to modify my local copy of stack.go to consider two Args to be similar if they have the same number of args (without looking at the args):

   func (a *Args) Similar(r *Args) bool {
     return a.Elided == r.Elided && len(a.Values) == len(r.Values)
   }
   

2. If a goroutine has been blocked for a long time on e..g a channel op or IO, it prints something like

   goroutine 134103 [chan send, 609 minutes]:
   

   or

   goroutine 134227 [IO wait, 747 minutes]:
   

   A difference in # of minutes causes two goroutines not to be deduped. My quick hack here was to modify my trace file with a quick s/\d+ minutes/123 minutes/. A more robust solution would be for panicparse to merge multiple values by replacing with * minutes (as with merged function params) or, if we want to be fancy, something like (12-49) minutes.

I wanted to bring these issues to your attention and see what you think a good solution might be. I'm not sure if, for other people in other contexts, having goroutines not be disambiguated for the above two reasons is useful. If so, perhaps there could be a flag for even more aggressive deduplication.

All tests TestAugment passes on the amd64 architecture, but some of the tests would fail on all other architectures.

For example, with the recent Debian builds of panicparse 2.2.2-1:

| Test | Architectures on which the test failed | | ---------- | ---------- | | TestAugment/0-local_function_doesn't_interfere | arm64, armel, armhf, ppc64el, riscv64 | | TestAugment/1-func | armel, armhf | | TestAugment/12-string | arm64, armel, armhf, ppc64el, riscv64 | | TestAugment/13-string_and_int | arm64, armel, armhf, ppc64el, riscv64 | | TestAugment/16-error_unnamed | arm64, armel, armhf, i386, mips64el, mipsel, ppc64el, riscv64, s390x | | TestAugment/24-argument_offsets_partially_too_large | armel, armhf, i386, mipsel | | TestAugment/25-argument_offsets_entirely_too_large | armel, armhf, i386, mipsel |

IIRC, the mapping from Debian architecture names to GOARCH are as follows:

• armel and armhf ⇒ GOARCH=arm
• i386 ⇒ GOARCH=386
• mips64el ⇒ GOARCH=mips64le
• mipsel ⇒ GOARCH=mipsle
• ppc64el ⇒ GOARCH=ppc64le

Screen capture of overview of Debian build status for panicparse (https://buildd.debian.org/status/package.php?p=panicparse):

Thanks!

Hello,

Interesting package. I think it's suitable for fuzzing. Proposing to get it running on Fuzzit. They give free service for open source.

They use go-fuzz. It's one of these genetic fuzzers, it instruments the binary and tries to traverse all the branches. A local run has built up a corpus of a couple hundred cases. No crashes so far. It seems to be in good shape.

Build will fail due to missing the API key. If you're interested setup is like this:

• In Fuzzit create the target panicparse.
• In Fuzzit settings grab an API key. In repo settings in Travis paste it to envvar FUZZIT_API_KEY.

This is a pretty minor thing, but it confused me enough to make me want to fix it, unless its a harder problem than I'm realizing.

Empty structs are decoded as ..., `` or sometimes as unknown.

I'd be happy to take a stab at this tonight if this is indeed a bug.

Examples:

//go:noinline
func Foo(x struct{}) {
	panic("test")
}

func TestFoo(t *testing.T) {
	Foo(struct{}{})
}

Results in:

1: running [Created by testing.(*T).Run @ testing.go:1238]
    testing testing.go:1143    tRunner.func1.2(*T(#1), func(#2))
    testing testing.go:1146    tRunner.func1(*T(#3))
            panic.go:965       panic(interface{}(#1))
    pp ppl_test.go:16 Foo()
    pp pp_test.go:20 TestFoo(*T(#3))
    testing testing.go:1193    tRunner(*T(#3), func(0x8952b8))
exit status 2

And

//go:noinline
func Foo(x chan struct{}) {
	panic("test")
}

func TestFoo(t *testing.T) {
	Foo(nil)
}

Results in:

1: running [Created by testing.(*T).Run @ testing.go:1238]
    testing testing.go:1143    tRunner.func1.2(*T(#1), func(#2))
    testing testing.go:1146    tRunner.func1(*T(#3))
            panic.go:965       panic(interface{}(#1))
    pp ppl_test.go:16 Foo(chan <unknown>(0x0))
    pp pp_test.go:20 TestFoo(*T(#3))
    testing testing.go:1193    tRunner(*T(#3), func(0x8952b8))
exit status 2

It would be cool if the tool supported an output format similar to "typical compilers", so that it could be easily fetched into vim's "quickfix" window, for easy jumping through the lines shown in the trace. For example:

1: running...
/foo/bar/baz/tool.go:123: (*whatever) FlavorText()
/foo/bar/baz/tool.go:55: (*whatever) Something()
/foo/bar/baz/main.go:22: CREATED BY: main()
5: ...
  ...
7: ...
  ...

When imported to vim with :cex system('panicparse -format=vim -only < foobar.log') | copen, this should make it trivial to jump between tool.go l.123, main.go l.22, etc., using :cn and :cp commands. (With properly configured :set errorformat=... in vim, the lines like 1: running... would still get displayed in the quickfix window, clearly separating the goroutines.)

edit: the Go oracle tool has it as: -format=plain