Stochastic flame graph profiler for Go programs

go-torch Build Status Coverage Status GoDoc

go-torch is deprecated, use pprof instead

As of Go 1.11, flamegraph visualizations are available in go tool pprof directly!

# This will listen on :8081 and open a browser.
# Change :8081 to a port of your choice.
$ go tool pprof -http=":8081" [binary] [profile]

If you cannot use Go 1.11, you can get the latest pprof tool and use it instead:

# Get the pprof tool directly
$ go get -u github.com/google/pprof

$ pprof -http=":8081" [binary] [profile]

Synopsis

Tool for stochastically profiling Go programs. Collects stack traces and synthesizes them into a flame graph. Uses Go's built in pprof library.

Example Flame Graph

Inception

Basic Usage

$ go-torch -h
Usage:
  go-torch [options] [binary] <profile source>

pprof Options:
  -u, --url=         Base URL of your Go program (default: http://localhost:8080)
  -s, --suffix=      URL path of pprof profile (default: /debug/pprof/profile)
  -b, --binaryinput= File path of previously saved binary profile. (binary profile is anything accepted by https://golang.org/cmd/pprof)
      --binaryname=  File path of the binary that the binaryinput is for, used for pprof inputs
  -t, --seconds=     Number of seconds to profile for (default: 30)
      --pprofArgs=   Extra arguments for pprof

Output Options:
  -f, --file=        Output file name (must be .svg) (default: torch.svg)
  -p, --print        Print the generated svg to stdout instead of writing to file
  -r, --raw          Print the raw call graph output to stdout instead of creating a flame graph; use with Brendan Gregg's flame graph perl script (see https://github.com/brendangregg/FlameGraph)
      --title=       Graph title to display in the output file (default: Flame Graph)
      --width=       Generated graph width (default: 1200)
      --hash         Colors are keyed by function name hash
      --colors=      Set color palette. Valid choices are: hot (default), mem, io, wakeup, chain, java,
                     js, perl, red, green, blue, aqua, yellow, purple, orange
      --hash         Graph colors are keyed by function name hash
      --cp           Graph use consistent palette (palette.map)
      --inverted     Icicle graph
Help Options:
  -h, --help         Show this help message

Write flamegraph using /debug/pprof endpoint

The default options will hit http://localhost:8080/debug/pprof/profile for a 30 second CPU profile, and write it out to torch.svg

$ go-torch
INFO[19:10:58] Run pprof command: go tool pprof -raw -seconds 30 http://localhost:8080/debug/pprof/profile
INFO[19:11:03] Writing svg to torch.svg

You can customize the base URL by using -u

$ go-torch -u http://my-service:8080/
INFO[19:10:58] Run pprof command: go tool pprof -raw -seconds 30 http://my-service:8080/debug/pprof/profile
INFO[19:11:03] Writing svg to torch.svg

Or change the number of seconds to profile using --seconds:

$ go-torch --seconds 5
INFO[19:10:58] Run pprof command: go tool pprof -raw -seconds 5 http://localhost:8080/debug/pprof/profile
INFO[19:11:03] Writing svg to torch.svg

Using pprof arguments

go-torch will pass through arguments to go tool pprof, which lets you take existing pprof commands and easily make them work with go-torch.

For example, after creating a CPU profile from a benchmark:

$ go test -bench . -cpuprofile=cpu.prof

# This creates a cpu.prof file, and the $PKG.test binary.

The same arguments that can be used with go tool pprof will also work with go-torch:

$ go tool pprof main.test cpu.prof

# Same arguments work with go-torch
$ go-torch main.test cpu.prof
INFO[19:00:29] Run pprof command: go tool pprof -raw -seconds 30 main.test cpu.prof
INFO[19:00:29] Writing svg to torch.svg

Flags that are not handled by go-torch are passed through as well:

$ go-torch --alloc_objects main.test mem.prof
INFO[19:00:29] Run pprof command: go tool pprof -raw -seconds 30 --alloc_objects main.test mem.prof
INFO[19:00:29] Writing svg to torch.svg

Integrating With Your Application

To add profiling endpoints in your application, follow the official Go docs here. If your application is already running a server on the DefaultServeMux, just add this import to your application.

import _ "net/http/pprof"

If your application is not using the DefaultServeMux, you can still easily expose pprof endpoints by manually registering the net/http/pprof handlers or by using a library like this one.

Installation

$ go get github.com/uber/go-torch

You can also use go-torch using docker:

$ docker run uber/go-torch -u http://[address-of-host] -p > torch.svg

Using -p will print the SVG to standard out, which can then be redirected to a file. This avoids mounting volumes to a container.

Get the flame graph script:

When using the go-torch binary locally, you will need the Flamegraph scripts in your PATH:

$ cd $GOPATH/src/github.com/uber/go-torch
$ git clone https://github.com/brendangregg/FlameGraph.git

Development and Testing

Install the Go dependencies:

$ go get github.com/Masterminds/glide
$ cd $GOPATH/src/github.com/uber/go-torch
$ glide install

Run the Tests

$ go test ./...
ok    github.com/uber/go-torch   0.012s
ok    github.com/uber/go-torch/graph   0.017s
ok    github.com/uber/go-torch/visualization 0.052s
Owner
Uber Archive
Uber's open source projects archive. Support and/or new releases are not available.
Uber Archive
Comments
  • Simplify go-torch: use raw pprof output, and use go-flags

    Simplify go-torch: use raw pprof output, and use go-flags

    Simplify go-torch:

    • use go-flags instead of cli since we only need flags, we don't have subcommands etc. This simplifies the code and lets us use structs to access the command line flags.
    • Use raw output from pprof rather, which contains every single sample that we can convert to call stacks for flame graph scripts
    • reduce use of interfaces, test the real concrete implementations by having tests invoke "go tool pprof" and creating fake flamegraph scripts in our PATH for tests etc
    • removed logrus to reduce dependencies since we don't really need a smart logger.

    A lot of logic has been moved around into 2 packages: pprof: for invoking pprof and parsing the raw output from it renderer: for invoking flamegraph scripts

    r: @sectioneight

  • Add dockerfile and Glide files

    Add dockerfile and Glide files

    Hi!

    I added a couple of changes to the project:

    1. A brand new Dockerfile, so people can use go-torch from within a docker container using docker run uber/go-torch go-torch ...
    2. I removed the old Godeps file and added Glide files as requested.
  • pprof: fix incorrect parsing for locations

    pprof: fix incorrect parsing for locations

    Fixes #61.

    Extrapolating from https://github.com/google/pprof/blob/master/profile/profile.go#L360 and https://github.com/uber/go-torch/issues/61 we can have locations of the following format:

    <id>: 0x<address> M=<mapping id>
    

    Note: This doesn't address all malformed inputs, malformed inputs that may have gotten through before will still do so. This patch only addresses #61.

    Added the raw pprof out as an additional test for one of our binaries that failed prior.

  • Drop all but first line during location parsing

    Drop all but first line during location parsing

    This is a fix but not really a good solution for #63. The problem arises because the raw format (at least in go1.9) can contain lines of the form

    [...]
       729: 0x4d6a395 github.com/cockroachdb/cockroach/pkg/sql.(*tableState).release /Users/tschottdorf/go/src/github.com/cockroachdb/cockroach/pkg/sql/lease.go:791 s=0
       730: 0x4021625 runtime.heapBits.next /usr/local/Cellar/go/1.9beta2/libexec/src/runtime/mbitmap.go:464 s=0
                 runtime.scanobject /usr/local/Cellar/go/1.9beta2/libexec/src/runtime/mgcmark.go:1162 s=0
    [...]
    

    and the last line is not parsed correctly. The way parsing is set up is line-by-line, so it's not straightforward to recover. Instead of a bigger refactor to try to address this, this change just drops these extra lines, and we should really implement #64.

    Note also that I changed some lines in the parsing test to match what is actually produced by go1.9 -- I'm not sure if the previous snippet was simply not true-to-original or whether perhaps the raw format has changed. Either way, using the raw format seems to invite regular breakage.

    Closes #63.

  • Improve help + flags, update README

    Improve help + flags, update README

    Flags now pass through unknown flags (like --alloc_objects) to go tool pprof, and are broken into multiple sections.

    Restructure the README into:

    • Usage
    • Integration
    • Installation
    • Development

    Can see the rendered output here: https://github.com/uber/go-torch/blob/readme/README.md

    Focus examples on more common usage, and remove less frequently used examples like outputting an SVG to stdout or outputting the raw flamegraph input.

  • Fix sample selection to not always use first index

    Fix sample selection to not always use first index

    Currently, go-torch always uses the first sample it finds in the profile. With 1.8, memory profiles always have inuse/alloc so we need to select the right sample, otherwise we're always showing alloc.

    This allows allows selection of a custom sample using sample_index for non-memory profiles.

    Fixes #52.

  • Add perl to Docker image, and improve README instructions

    Add perl to Docker image, and improve README instructions

    Fixes "no such file or directory" error from #41

    @sectioneight: once we merge this, can you start a new build. We should look into automating the builds as well.

    cc @calavera

  • Allow to load the pprof raw data from a file.

    Allow to load the pprof raw data from a file.

    This change allows to analyze historic data without having to run pprof in real time. It's also useful when you cannot connect go-torch to a production server but the server has pprof installed.

    It adds the flag load to the CLI. This flag takes the file path to raw data. Go-torch tries to run the profiler as usual if this flag is empty.

    PS: Thanks for this project, it has already saved me lots of time.

    Signed-off-by: David Calavera [email protected]

  • pprof requires binary as well as pprof output when using files.

    pprof requires binary as well as pprof output when using files.

    from pprof usage: usage: pprof [options] [binary] ...

    require a binaryname arg when using binaryfile. verified this with local -cpuprofile output from tests.

    r @sandlerben

  • Fix pprof parsing when there's more than 2 columns

    Fix pprof parsing when there's more than 2 columns

    Previously, the pprof parser assumed there were 2 columns, and stored them as "samples" and "total". Instead parse all values and store columns which we can use in future.

    Fixes #51 but I noticed a new issue with how we select the sample to display: #52

  • Support pprof arg format in go-torch

    Support pprof arg format in go-torch

    pprof is usually run with either:

    go tool pprof [binary-name] [binary-file]
    

    or

    go tool pprof [url]
    

    This change makes it so go tool pprof can be replaced with go-torch and it just works. We support --seconds as well as --time (for backwards compatibility).

  • Improve instructions on fetching flamegraph scripts

    Improve instructions on fetching flamegraph scripts

    The first time I set this up on a new machine it pointed me to the repo, but didn't say which files I needed. So it took some trial and error before I got it working. We should be explicit in what we use.

Nats-subject-profiler - NATS Subject Profiler With Golang

NATS Subject Profiler Example Connect it to the demo NATS server. nats-subject-p

Feb 7, 2022
A highly-performant command runtime server written in Go for Flame 🔥

Flame Command Runtime A highly-performant command runtime server written in Go for Flame ?? Flame Command Runtime(FCR) is powerful, highly customizabl

Aug 29, 2021
FlameScope is a visualization tool for exploring different time ranges as Flame Graphs.
FlameScope is a visualization tool for exploring different time ranges as Flame Graphs.

FlameScope FlameScope is a visualization tool for exploring different time ranges as Flame Graphs, allowing quick analysis of performance issues such

Dec 27, 2022
JSONL graph tools - Graph is represented as JSONL of nodes and edges.

JSONL graph tools - Graph is represented as JSONL of nodes and edges.

Sep 27, 2022
The DGL Operator makes it easy to run Deep Graph Library (DGL) graph neural network training on Kubernetes

DGL Operator The DGL Operator makes it easy to run Deep Graph Library (DGL) graph neural network distributed or non-distributed training on Kubernetes

Dec 19, 2022
🚀 fgprof is a sampling Go profiler that allows you to analyze On-CPU as well as Off-CPU (e.g. I/O) time together.
🚀 fgprof is a sampling Go profiler that allows you to analyze On-CPU as well as Off-CPU (e.g. I/O) time together.

?? fgprof - The Full Go Profiler fgprof is a sampling Go profiler that allows you to analyze On-CPU as well as Off-CPU (e.g. I/O) time together. Go's

Dec 31, 2022
🚀Statsview is a real-time Golang runtime stats visualization profiler
🚀Statsview is a real-time Golang runtime stats visualization profiler

Statsview is a real-time Golang runtime stats visualization profiler. It is built top on another open-source project, go-echarts, which helps statsview to show its graphs on the browser.

Dec 29, 2022
System resource usage profiler tool which regularly takes snapshots of the memory and CPU load of one or more running processes so as to dynamically build up a profile of their usage of system resources.
System resource usage profiler tool which regularly takes snapshots of the memory and CPU load of one or more running processes so as to dynamically build up a profile of their usage of system resources.

Vegeta is a system resource usage tracking tool built to regularly take snapshots of the memory and CPU load of one or more running processes, so as to dynamically build up a profile of their usage of system resources.

Jan 16, 2022
Progress OpenEdge Profiler data parsing to OpenTracing format

openedge-profiler-parser Progress OpenEdge Profiler data parsing to OpenTracing format. Prerequisites In order to RUN you will be enough with Docker:

Nov 9, 2021
A simple application to show how to use dd-trace-go's tracer and profiler.

dd-trace-go-demo A simple application to show how to use dd-trace-go's tracer and profiler. Usage To run this demo application, simply clone this repo

Oct 8, 2022
Metrics collector and ebpf-based profiler for C, C++, Golang, and Rust

Apache SkyWalking Rover SkyWalking Rover: Metrics collector and ebpf-based profiler for C, C++, Golang, and Rust. Documentation Official documentation

Jan 6, 2023
BTree provides a simple, ordered, in-memory data structure for Go programs.

BTree implementation for Go This package provides an in-memory B-Tree implementation for Go, useful as an ordered, mutable data structure. The API is

Dec 30, 2022
webbased debugging for Go programs
webbased debugging for Go programs

Hopwatch, a debugging tool for Go Hopwatch is a simple tool in HTML5 that can help debug Go programs. It works by communicating to a WebSockets based

Nov 26, 2022
An embeddable implementation of the Ngaro Virtual Machine for Go programs

Ngaro Go Overview This is an embeddable Go implementation of the Ngaro Virtual Machine. This repository contains the embeddable virtual machine, a rud

Dec 3, 2022
Read and write Netpbm images from Go programs

netpbm Introduction netpbm is a package for the Go programming language that implements image decoders and encoders for the Netpbm image formats. The

Dec 29, 2022
Simple logger for Go programs. Allows custom formats for messages.
Simple logger for Go programs. Allows custom formats for messages.

go-logger A simple go logger for easy logging in your programs. Allows setting custom format for messages. Preview Install go get github.com/apsdehal/

Dec 17, 2022
Example programs for the Gio project.

Gio Examples Example programs for the Gio project. Issues File bugs and TODOs through the issue tracker or send an email to ~eliasnaur/[email protected].

Dec 20, 2022
Simple project to demonstrate the loading of eBPF programs via florianl/go-tc.

tc-skeleton Simple project to demonstrate the loading of eBPF programs via florianl/go-tc.

Dec 23, 2022
Library to work with eBPF programs from Go

Go eBPF A nice and convenient way to work with eBPF programs / perf events from Go. Requirements Go 1.10+ Linux Kernel 4.15+ Supported eBPF features e

Dec 29, 2022
7 days golang programs from scratch (web framework Gee, distributed cache GeeCache, object relational mapping ORM framework GeeORM, rpc framework GeeRPC etc) 7天用Go动手写/从零实现系列

7 days golang programs from scratch README 中文版本 7天用Go从零实现系列 7天能写什么呢?类似 gin 的 web 框架?类似 groupcache 的分布式缓存?或者一个简单的 Python 解释器?希望这个仓库能给你答案

Jan 5, 2023