pprofutils pprofutils is a swiss army knife for pprof files.

documentation ci test status

pprofutils

pprofutils is a swiss army knife for pprof files. You can use it as a command line utility or as a free web service.

Install

pprofutils requires Go 1.16 and can be installed like this:

go install github.com/felixge/pprofutils/v2/cmd/pprofutils@latest

Alternatively you can use it as a free web service hosted at https://pprof.to.

Utilities

anon

Takes a pprof profile and anonymizes it by replacing pkg, file and function names with human readable hashes. The whitelist can be used to prevent certain packages from being anonymized.

TODO: Ignore all stdlib packages by default and maybe also popular OSS libs.

The input and output file default to "-" which means stdin or stdout.

Use anon utility via cli

pprofutils anon [-whitelist=
   
    ] 
     
    

FLAGS:
  -whitelist=^runtime;^net;^encoding Semicolon separated pkg name regex list

   

Use anon utility via web service

curl --data-binary @ 'pprof.to/anon?whitelist=^runtime;^net;^encoding' > 

Example 1: Anonymize a CPU profile

pprofutils anon examples/anon.in.pprof examples/anon.out.pprof
# or
curl --data-binary @examples/anon.in.pprof pprof.to/anon > examples/anon.out.pprof

Converts the profile examples/anon.in.pprof that looks like this:

Into a new profile examples/anon.out.pprof that looks like this:

avg

Takes a block or mutex profile and creates a profile that contains the average time per contention by dividing the nanoseconds or value in the profile by the sample count value.

TODO: Support memory profiles.

The input and output file default to "-" which means stdin or stdout.

Use avg utility via cli

pprofutils avg  

Use avg utility via web service

curl --data-binary @ 'pprof.to/avg' > 

Example 1: Convert block profile to avg time

pprofutils avg examples/avg.in.pprof examples/avg.out.pprof
# or
curl --data-binary @examples/avg.in.pprof pprof.to/avg > examples/avg.out.pprof

Converts the profile examples/avg.in.pprof that looks like this:

Into a new profile examples/avg.out.pprof that looks like this:

folded

Converts pprof to Brendan Gregg's folded text format and vice versa. The input format is automatically detected and used to determine the output format.

The input and output file default to "-" which means stdin or stdout.

Use folded utility via cli

pprofutils folded [-headers]  

FLAGS:
  -headers=false Add header column for each sample type

Use folded utility via web service

curl --data-binary @ 'pprof.to/folded?headers=false' > 

Example 1: Convert folded text to pprof

pprofutils folded examples/folded.in.txt examples/folded.out.pprof
# or
curl --data-binary @examples/folded.in.txt pprof.to/folded > examples/folded.out.pprof

Converts examples/folded.in.txt with the following content:

main;foo 5
main;foo;bar 3
main;foobar 4

Into a new profile examples/folded.out.pprof that looks like this:

Example 2: Convert pprof to folded text

pprofutils folded examples/folded.in.pprof examples/folded.out.txt
# or
curl --data-binary @examples/folded.in.pprof pprof.to/folded > examples/folded.out.txt

Converts the profile examples/folded.in.pprof that looks like this:

Into a new folded text file examples/folded.out.txt that looks like this:

main;foo 5
main;foobar 4
main;foo;bar 3

json

Converts from pprof to json and vice vera. The input format is automatically detected and used to determine the output format.

The input and output file default to "-" which means stdin or stdout.

Use json utility via cli

pprofutils json  

Use json utility via web service

curl --data-binary @ 'pprof.to/json' > 

Example 1: Convert pprof to json

pprofutils json examples/json.in.pprof examples/json.out.json
# or
curl --data-binary @examples/json.in.pprof pprof.to/json > examples/json.out.json

See examples/json.in.pprof and examples/json.out.json for more details.

Example 2: Convert json to pprof

pprofutils json examples/json.in.json examples/json.out.pprof
# or
curl --data-binary @examples/json.in.json pprof.to/json > examples/json.out.pprof

See examples/json.in.json and examples/json.out.pprof for more details.

labelframes

Adds virtual root frames for the each value of the selected pprof label. This is useful to visualize label values in a flamegraph.

The input and output file default to "-" which means stdin or stdout.

Use labelframes utility via cli

pprofutils labelframes -label=

Use labelframes utility via web service

curl --data-binary @ 'pprof.to/labelframes?label=mylabel' > 

Example 1: Add root frames for pprof label values

pprofutils labelframes examples/labelframes.in.pprof examples/labelframes.out.pprof
# or
curl --data-binary @examples/labelframes.in.pprof pprof.to/labelframes > examples/labelframes.out.pprof

Converts the profile examples/labelframes.in.pprof that looks like this:

Into a new profile examples/labelframes.out.pprof that looks like this:

raw

Converts pprof to the same text format as go tool pprof -raw.

The input and output file default to "-" which means stdin or stdout.

Use raw utility via cli

pprofutils raw  

Use raw utility via web service

curl --data-binary @ 'pprof.to/raw' > 

Example 1: Convert pprof to raw

pprofutils raw examples/raw.in.pprof examples/raw.out.txt
# or
curl --data-binary @examples/raw.in.pprof pprof.to/raw > examples/raw.out.txt

See examples/raw.in.pprof and examples/raw.out.txt for more details.

Use Cases

Convert linux perf profiles to pprof

Convert a Linux perf.data profile to pprof, via Brendan Gregg's stackcollapse-perf.pl script:

perf script | stackcollapse-perf.pl | pprofutils folded > perf.pprof

License

pprofutils is licensed under the MIT License.

Owner
Comments
  • Add an extra end-to-end example with stackcollapse

    Add an extra end-to-end example with stackcollapse

    And add some words around 'stackcollapse' to help people find this repo if they're looking for scripts to deal with the output of the stackcollapse-* scripts.

  • Support conversion of jemalloc format to pprof format

    Support conversion of jemalloc format to pprof format

    pprof does not support jemalloc plain text format, so here we add support for convert jemalloc plain text format to pprof format.

    This is similar to https://github.com/google/pprof/pull/690, however by doing it inside pprofutils, we avoid having to change pprof or jemalloc.

    Signed-off-by: Daniel Dao [email protected]

  • Support outputting line numbers in folded format.

    Support outputting line numbers in folded format.

    Hey Felix, thanks for the nice toolset here :wave:

    I've recently needed to do line-granularity deduplication work between 2 profiles and supporting line numbers on the folded output was extremely helpful. I imagine other people might want it as well and, as such, thought having a flag for it would be a good improvement 😄

    Proof that this works (apart from included unit test):

    ❯ pprofutils folded -line_numbers=true go.alloc.pb | head -n 3
    github.com/DataDog/dd-go/trace/intake/common.(*Resolver).resolveStats.func1:547;github.com/DataDog/dd-go/trace/agent/model.TagSet.Key:74;github.com/DataDog/dd-go/trace/agent/model.Tag.String:20 44631378
    github.com/DataDog/dd-go/trace/intake/common.(*Resolver).resolveStats.func1:494;github.com/DataDog/dd-go/trace/intake/common.(*Resolver).resolveTagSet:422;github.com/DataDog/dd-go/trace/agent/model.SplitTag:26;strings.SplitN:282;strings.genSplit:256 27804496
    github.com/DataDog/dd-go/trace/intake/common.(*Resolver).resolveStats.func1:507;github.com/DataDog/dd-go/trace/agent/model.SplitTag:26;strings.SplitN:282;strings.genSplit:256 27165501
    
    ❯ pprofutils folded -line_numbers=false go.alloc.pb | head -n 3
    github.com/DataDog/dd-go/trace/intake/common.(*Resolver).resolveStats.func1;github.com/DataDog/dd-go/trace/agent/model.TagSet.Key;github.com/DataDog/dd-go/trace/agent/model.Tag.String 60852033
    github.com/DataDog/dd-go/trace/intake/common.(*Resolver).resolveStats.func1;github.com/DataDog/dd-go/trace/intake/common.(*Resolver).resolveTagSet;github.com/DataDog/dd-go/trace/agent/model.SplitTag;strings.SplitN;strings.genSplit 41354477
    github.com/DataDog/dd-go/trace/intake/common.(*Resolver).resolveStats.func1;github.com/DataDog/dd-go/trace/agent/model.SplitTag;strings.SplitN;strings.genSplit 39797950
    

    And that extra granularity does not somehow lead to wrong values:

    ❯ pprofutils folded -line_numbers=false go.alloc.pb | awk '{ s += $2 } END { print s }' 
    2363251702
    
    ❯ pprofutils folded -line_numbers=true go.alloc.pb | awk '{ s += $2 } END { print s }'
    2363251702
    
  • Unknown Errror with JSON File

    Unknown Errror with JSON File

    I am attempting to convert the below JSON into a PPROF file. However, pprofutils returns the below error message:

    input format is neither pprof nor json

    Is there a way to identify what in the file is causing this error? I have used multiple json validators to confirm the format is valid.

    single_stack_trace_ex.txt

  • Allow legacy profile to have space in stack

    Allow legacy profile to have space in stack

    First of all, thanks for the tool. It's awesome !

    Now to the PR :) In some of our apps, we generate a folded text that has space in it. For example:

    doing_something;JIT Compiler 1
    

    This breaks current legacy converter because it's using string.Split. The current PR changes it so that we parse the line with the number of expected whitespace from last, with number of whitespace equals to len length of SampleType.

    This solves the problem for us, and probably solves https://github.com/felixge/pprofutils/issues/3 as well.

  • "Bad line" error on an Android simpleperf trace

    Came across this trying to convert an android simpleperf trace to pprof:

    I'll note the full repo steps here in case they're useful; they probably aren't though; the error message is probably sufficient:

    $ git clone https://android.googlesource.com/platform/system/extras
    $ cd extras/simpleperf/scripts
    $ ./app_profiler.py -p com.google.android.apps.maps -r "-f 4000 -g -e cpu-clock --trace-offcpu --duration 60" --activity com.google.android.maps.MapsActivity --skip_collect_binaries
    $ ./report_sample.py ~/simpleperf/perf.data | ~/FlameGraph/stackcollapse-perf.pl | ~/go/bin/text2pprof
    

    Gives this error:

    bad line: 6: "Binder:23281_1;__start_thread;__pthread_start;thread_data_t::trampoline;android::AndroidRuntime::javaThreadShell;android::Thread::_threadLoop;android::PoolThread::threadLoop;android::IPCThreadState::joinThreadPool;android::IPCThreadState::executeCommand;android::BBinder::transact;JavaBBinder::onTransact;_JNIEnv::CallBooleanMethod;art::JNI<false>::CallBooleanMethodV;art::JValue art::InvokeVirtualOrInterfaceWithVarArgs<art::ArtMethod*> 6"
    

    This happens on line 6... let's look at the first 10 lines:

    ./report_sample.py ~/simpleperf/perf.data | ~/FlameGraph/stackcollapse-perf.pl | head
    Binder:23281_1;__start_thread;__pthread_start;thread_data_t::trampoline;android::AndroidRuntime::javaThreadShell;android::Thread::_threadLoop;android::PoolThread::threadLoop;android::IPCThreadState::joinThreadPool 11
    Binder:23281_1;__start_thread;__pthread_start;thread_data_t::trampoline;android::AndroidRuntime::javaThreadShell;android::Thread::_threadLoop;android::PoolThread::threadLoop;android::IPCThreadState::joinThreadPool;@plt 6
    Binder:23281_1;__start_thread;__pthread_start;thread_data_t::trampoline;android::AndroidRuntime::javaThreadShell;android::Thread::_threadLoop;android::PoolThread::threadLoop;android::IPCThreadState::joinThreadPool;@plt;work_pending;do_notify_resume;__schedule;__schedule 3
    Binder:23281_1;__start_thread;__pthread_start;thread_data_t::trampoline;android::AndroidRuntime::javaThreadShell;android::Thread::_threadLoop;android::PoolThread::threadLoop;android::IPCThreadState::joinThreadPool;android::IPCThreadState::executeCommand 10
    Binder:23281_1;__start_thread;__pthread_start;thread_data_t::trampoline;android::AndroidRuntime::javaThreadShell;android::Thread::_threadLoop;android::PoolThread::threadLoop;android::IPCThreadState::joinThreadPool;android::IPCThreadState::executeCommand;android::BBinder::transact 1
    Binder:23281_1;__start_thread;__pthread_start;thread_data_t::trampoline;android::AndroidRuntime::javaThreadShell;android::Thread::_threadLoop;android::PoolThread::threadLoop;android::IPCThreadState::joinThreadPool;android::IPCThreadState::executeCommand;android::BBinder::transact;JavaBBinder::onTransact;_JNIEnv::CallBooleanMethod;art::JNI<false>::CallBooleanMethodV 2
    Binder:23281_1;__start_thread;__pthread_start;thread_data_t::trampoline;android::AndroidRuntime::javaThreadShell;android::Thread::_threadLoop;android::PoolThread::threadLoop;android::IPCThreadState::joinThreadPool;android::IPCThreadState::executeCommand;android::BBinder::transact;JavaBBinder::onTransact;_JNIEnv::CallBooleanMethod;art::JNI<false>::CallBooleanMethodV;art::JValue art::InvokeVirtualOrInterfaceWithVarArgs<art::ArtMethod*> 6
    Binder:23281_1;__start_thread;__pthread_start;thread_data_t::trampoline;android::AndroidRuntime::javaThreadShell;android::Thread::_threadLoop;android::PoolThread::threadLoop;android::IPCThreadState::joinThreadPool;android::IPCThreadState::executeCommand;android::BBinder::transact;JavaBBinder::onTransact;_JNIEnv::CallBooleanMethod;art::JNI<false>::CallBooleanMethodV;art::JValue art::InvokeVirtualOrInterfaceWithVarArgs<art::ArtMethod*>;art_quick_invoke_stub;android.os.Binder.execTransact 2
    Binder:23281_1;__start_thread;__pthread_start;thread_data_t::trampoline;android::AndroidRuntime::javaThreadShell;android::Thread::_threadLoop;android::PoolThread::threadLoop;android::IPCThreadState::joinThreadPool;android::IPCThreadState::executeCommand;android::BBinder::transact;JavaBBinder::onTransact;_JNIEnv::CallBooleanMethod;art::JNI<false>::CallBooleanMethodV;art::JValue art::InvokeVirtualOrInterfaceWithVarArgs<art::ArtMethod*>;art_quick_invoke_stub;android.os.Binder.execTransact;android.os.Binder.execTransactInternal 2
    Binder:23281_1;__start_thread;__pthread_start;thread_data_t::trampoline;android::AndroidRuntime::javaThreadShell;android::Thread::_threadLoop;android::PoolThread::threadLoop;android::IPCThreadState::joinThreadPool;android::IPCThreadState::executeCommand;android::BBinder::transact;JavaBBinder::onTransact;_JNIEnv::CallBooleanMethod;art::JNI<false>::CallBooleanMethodV;art::JValue art::InvokeVirtualOrInterfaceWithVarArgs<art::ArtMethod*>;art_quick_invoke_stub;android.os.Binder.execTransact;android.os.Binder.execTransactInternal;ExecuteNterpImpl 1
    

    At a guess, I'd say the problem is probably the space in between JValue art.

    It's very possible that I'm holding this wrong and there shouldn't be spaces in these lines according to the format? However perhaps there is a way to 'greedily' consume forward till we get to the numbers, perhaps?

Split text files into gzip files with x lines

hakgzsplit split lines of text into multiple gzip files

Jun 21, 2022
Easily create Go files from stub files

go-stubs Easily create .go files from stub files in your projects. Usage go get github.com/nwby/go-stubs Create a stub file: package stubs type {{.Mo

Jan 27, 2022
app-services-go-linter plugin analyze source tree of Go files and validates the availability of i18n strings in *.toml files

app-services-go-linter app-services-go-linter plugin analyze source tree of Go files and validates the availability of i18n strings in *.toml files. A

Nov 29, 2021
Compute message digest for large files in Go

checksum test coverage Compute message digest, like MD5 and SHA256, in golang for potentially large files. Usage package main import ( "fmt" "githu

Dec 28, 2022
copy files for humans

Go-Decent-Copy go-decent-copy provides a copy file for humans Usage package main import "github.com/hugocarreira/go-decent-copy" func main() { e

Sep 26, 2022
Golang wrapper for Exiftool : extract as much metadata as possible (EXIF, ...) from files (pictures, pdf, office documents, ...)

go-exiftool go-exiftool is a golang library that wraps ExifTool. ExifTool's purpose is to extract as much metadata as possible (EXIF, IPTC, XMP, GPS,

Dec 28, 2022
Load GTFS files in golang

go-gtfs Load GTFS files in Go. The project is in maintenance mode. It is kept compatible with changes in the Go ecosystem but no new features will be

Dec 5, 2022
An implementation of the FileSystem interface for tar files.

TarFS A wrapper around tar.Reader. Implements the FileSystem interface for tar files. Adds an Open method, that enables reading of file according to i

Sep 26, 2022
QueryCSV enables you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to a CSV file
QueryCSV enables you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to a CSV file

QueryCSV enable you to load CSV files and manipulate them using SQL queries then after you finish you can export the new values to CSV file

Dec 22, 2021
A Go io/fs filesystem implementation for reading files in a Github gists.

GistFS GistFS is an io/fs implementation that enables to read files stored in a given Gist. Requirements This module depends on io/fs which is only av

Oct 14, 2022
Create ePub files from URLs

url2epub Create ePub files from URLs Overview The root directory provides a Go library that creates ePub files out of URLs, with limitations.

Nov 5, 2022
🏵 Gee is tool of stdin to each files and stdout
🏵 Gee is tool of stdin to each files and stdout

Gee is tool of stdin to each files and stdout. It is similar to the tee command, but there are more functions for convenience. In addition, it was written as go. which provides output to stdout and files.

Nov 17, 2022
Golang PDF library for creating and processing PDF files (pure go)

UniPDF - PDF for Go UniDoc UniPDF is a PDF library for Go (golang) with capabilities for creating and reading, processing PDF files. The library is wr

Dec 28, 2022
Format /etc/fstab files.
Format /etc/fstab files.

Format /etc/fstab files. Features and limitations Can format /etc/fstab files. Will use 2 spaces between all fields, if they are of equal length. The

Dec 3, 2022
Run a command when files change

Reflex Reflex is a small tool to watch a directory and rerun a command when certain files change. It's great for automatically running compile/lint/te

Dec 29, 2022
Go (golang) library for reading and writing XLSX files.

XLSX Introduction xlsx is a library to simplify reading and writing the XML format used by recent version of Microsoft Excel in Go programs. Current s

Jan 5, 2023
Golang library for reading and writing Microsoft Excel™ (XLSX) files.
Golang library for reading and writing Microsoft Excel™ (XLSX) files.

Excelize Introduction Excelize is a library written in pure Go providing a set of functions that allow you to write to and read from XLSX / XLSM / XLT

Dec 31, 2022
Easily create & extract archives, and compress & decompress files of various formats

archiver Introducing Archiver 3.1 - a cross-platform, multi-format archive utility and Go library. A powerful and flexible library meets an elegant CL

Jan 3, 2023
Embed files into a Go executable

statik statik allows you to embed a directory of static files into your Go binary to be later served from an http.FileSystem. Is this a crazy idea? No

Jan 6, 2023