Go Dependency Analysis toolkit

Goda

Goda is a Go dependency analysis toolkit. It contains tools to figure out what your program is using.

Note: the exact syntax of the command line arguments has not yet been finalized. So expect some changes to it.

Cool things it can do:

# All of the commands should be run in the cloned repository.
git clone https://github.com/loov/goda && cd goda

# draw a graph of packages in github.com/loov/goda
goda graph "github.com/loov/goda/..." | dot -Tsvg -o graph.svg

# draw a dependency graph of github.com/loov/goda and dependencies
goda graph -cluster -short "github.com/loov/goda:all" | dot -Tsvg -o graph.svg

# list direct dependencies of github.com/loov/goda
goda list "github.com/loov/goda/...:import"

# list dependency graph that reaches flag package, including std
goda graph -std "reach(github.com/loov/goda/...:all, flag)" | dot -Tsvg -o graph.svg

# list packages shared by github.com/loov/goda/pkgset and github.com/loov/goda/cut
goda list "shared(github.com/loov/goda/pkgset:all, github.com/loov/goda/cut:all)""

# list packages that are only imported for tests
goda list "test=1(github.com/loov/goda/...:all) - test=0(github.com/loov/goda/...:all)"

# list packages that are imported with `purego` tag
goda list -std "purego=1(github.com/loov/goda/...:all)"

# list packages that are imported for windows and not linux
goda list "goos=windows(github.com/loov/goda/...:all) - goos=linux(github.com/loov/goda/...:all)"

# list how much memory each symbol in the final binary is taking
goda weight -h $GOPATH/bin/goda

# show the impact of cutting a package
goda cut ./...:all

# print dependency tree of all sub-packages
goda tree ./...:all

# print stats while building a go program
go build -a --toolexec "goda exec" .

Maybe you noticed that it's using some weird symbols on the command-line while specifying packages. They allow for more complex scenarios.

The basic syntax is that you can specify multiple packages:

goda list github.com/loov/goda/... github.com/loov/qloc

By default it will select all the specific packages. You can select the package dependencies with :import and both of them with :all:

goda list github.com/loov/goda/...:import:all
goda list github.com/loov/goda/...:all

You can also do basic arithmetic with these sets. For example, if you wish to ignore all golang.org/x/tools dependencies:

goda list github.com/loov/goda/...:all - golang.org/x/tools/...

Graph example

Here's an example output for:

git clone https://github.com/loov/goda && cd goda
goda graph github.com/loov/goda/... | dot -Tsvg -o graph.svg

github.com/loov/goda dependency graph

How it differs from go list or go mod

go list and go mod are tightly integrated with Go and can answer simple queries with compatibility. They also serves as good building blocks for other tools.

goda is intended for more complicated queries and analysis. Some of the features can be reproduced by format flags and scripts. However, this library aims to make even complicated analysis fast.

Also, goda can be used together with go list and go mod.

Comments
  • Support for vendored packages

    Support for vendored packages

    I really enjoy goda! It's been very helpful on debugging. Now I want to use it as part of our CI to build and test the application when the relevant packages have changed.

    Our CI is air gapped and all the packages vendored, but when I call goda list, I see it's trying to fetch the packages from the internet.

    Is there a way to enable vendored package in goda?

  • Include version information on packages/modules

    Include version information on packages/modules

    I am attempting to use goda to produce the same form of output as go mod graph, for use in an application that parses the latter. goda graph -type edges comes close, save for the lack of version suffixes. There does not appear to be a format string argument that will provide this.

    It would be helpful for version information to at least be available (as a label-formatting directive) when generating dependency output.

  • graph: repository and module clustering

    graph: repository and module clustering

    I don't find the clusters chosen by goda -cluster map to concepts in Go. I think clustering based on repository and module membership provides a more useful visualization. Here's a rough draft of some changes to make goda -cluster work that way. Is this something you would consider adding to goda if it was polished up a bit more?

  • #62 Only add node if it doesn't contain errors

    #62 Only add node if it doesn't contain errors

    Thanks for the great tool.

    I did run across a null pointer dereference error when attempting to create a graph with the -cluster command, as described by someone else in #62

    I noticed that when LoadNode() is called, it sets an error if the repo cannot be loaded. If the error is set, it means the repo is not set, so anything attempting to access the node repo is then crashing. Rather than checking for nil values in specific places, I opted to no longer process that node at all.

  • Add

    Add "goda why" or similar command

    I have very often wanted to run a command to see why a dependency is required. This could be similar to go mod why, although that command is lacking a bit of functionality.

    Essentially, given an import tree of

    .
     ├  a
       ├ b
       ├ c
     ├ d
       ├  c 
     ├  c 
    

    I would like a command like: goda why . c I would expect:

    $ goda why . c
    . -> a -> c
    . -> d -> c
    . -> c
    

    From this, I can see all of the places I need to clean up usages of c if I want to drop the dependency.

    I have done this exact thing before using a bunch of ad-hoc goda commands, resulting in out binary sizes being cut in half, so I think it would be pretty useful. It feels like all of the information is there, but not presentable in this way. Its also possible I just haven't found the right expressions to represent this.

    Alternatives:

    • Modify graph/tree in some way to allow filtering out packages not in the path of c. For example, the above tree would be filtered to:
    .
     ├  a
       ├ c
     ├ d
       ├  c 
     ├  c 
    

    Here the tree is pretty much the same, but in our real repo the tree is 12k lines so its not usable for this use case. The same with the graph - its too big to read without filtering

  • command `nm` unintuitive

    command `nm` unintuitive

    From the list of subcommands, nm is the one which could have a more intuitive name. Depending on future developments I would recommend binary or memory as a replacement.

  • panic: runtime error: invalid memory address or nil pointer dereference

    panic: runtime error: invalid memory address or nil pointer dereference

    I'm not sure what other details I can provide that would be helpful since the repo is private but happy to provide whatever additional details I can.

    $ goda graph -cluster -short "github.com/me/app:all" | dot -Tsvg -o graph.svg

    panic: runtime error: invalid memory address or nil pointer dereference
    [signal SIGSEGV: segmentation violation code=0x1 addr=0x18 pc=0x1408ce1]
    
    goroutine 1 [running]:
    github.com/loov/goda/internal/pkgtree.(*Tree).NodeRepo(...)
    	/Users/me/go/pkg/mod/github.com/loov/[email protected]/internal/pkgtree/tree.go:75
    github.com/loov/goda/internal/pkgtree.From(0xc000076000, 0xb061, 0x5c0, 0xee)
    	/Users/me/go/pkg/mod/github.com/loov/[email protected]/internal/pkgtree/tree.go:26 +0x121
    github.com/loov/goda/internal/graph.(*Dot).WriteClusters(0xc000154980, 0xc000076000, 0x0, 0x0)
    	/Users/me/go/pkg/mod/github.com/loov/[email protected]/internal/graph/dot.go:131 +0x76
    github.com/loov/goda/internal/graph.(*Dot).Write(0xc000154980, 0xc000076000, 0xc0008d2270, 0x1)
    	/Users/me/go/pkg/mod/github.com/loov/[email protected]/internal/graph/dot.go:106 +0x3b
    github.com/loov/goda/internal/graph.(*Command).Execute(0xc000154640, 0x157f370, 0xc00011a000, 0xc00012e660, 0x0, 0x0, 0x0, 0x0)
    	/Users/me/go/pkg/mod/github.com/loov/[email protected]/internal/graph/cmd.go:136 +0x339
    github.com/google/subcommands.(*Commander).Execute(0xc000130200, 0x157f370, 0xc00011a000, 0x0, 0x0, 0x0, 0xffffffff)
    	/Users/me/go/pkg/mod/github.com/google/[email protected]/subcommands.go:209 +0x347
    main.main()
    	/Users/me/go/pkg/mod/github.com/loov/[email protected]/main.go:38 +0x39d
    
  • Support build tags

    Support build tags

    Currently it seems build tags are not respected. I would like files like // +build !bloat to be ignored, since they will not end up in my final binaries anyways. Specifically with graph, but likely others apply as well

  • make it easier to select tests

    make it easier to select tests

    Currently to include test packages, you can use test=1(something) but that's awkward to type, not that easy to remember and requires shell quotes.

    I find myself needing this qualifier a lot, because test code is a large part of any repository and a big part of refactoring calculations.

    How about using a :test qualifier instead?

    So, for example, ./...:test would include all the test code in ./....

    ./...:test:all would include all dependencies of all packages and tests below ..

  • pkgset: breaking, make default behavior :root

    pkgset: breaking, make default behavior :root

    This change:

    • removes :root, this is the default behavior
    • removes :noroot, and replaces it with :deps
    • adds :all, to get the previous behavior

    When using the expression syntax, the first usage is often to target all the roots and the dependenices... however most reduction need to use :root. Effectively making the expressions longer than they need to be.

    For a comparison:

    X - golang.org/x/tools/...:root - github.com/xyz/...:root

    After this change:

    X:all - golang.org/x/tools/... - github.com/xyz/...

    Arguably, the second one is easier to read and understand.

    CC @rogpeppe

  • Override documentation URL

    Override documentation URL

    Feature Request We run our own internal GoDoc. Being able to point at that instead of pkg.go.dev would make this package usable in our GoDoc.

    What's currently happening Clicking a node yields documentation for pkg.go.dev which only serves public packages

    What you want to happen Be able to override the pkg.go.dev URL.

  • internal/pkgset: avoid loading all std packages

    internal/pkgset: avoid loading all std packages

    Currently the code loads all the std library packages, which takes a bunch of time. It's only used to remove std library packages from the output.

    https://github.com/loov/goda/blob/v0.5.3/internal/pkgset/std.go#L13

    Instead of loading the std packages separately, write a func that detects based on packages.Package whether it's a std library package or not.

  • avoid test packages in cut output

    avoid test packages in cut output

    When I run goda cut, I see a lot of test package results like this:

    github.com/influxdata/idpe/client/v2 [github.com/influxdata/idpe/client/v2.test]                                                                            in:0     pkgs:1   size:45.6KB    loc:1499
    github.com/influxdata/idpe/client/v2.test                                                                                                                   in:0     pkgs:1   size:2.9KB     loc:58
    github.com/influxdata/idpe/client/v2_test [github.com/influxdata/idpe/client/v2.test]                                                                       in:0     pkgs:1   size:6.0KB     loc:234
    github.com/influxdata/idpe/client/v2                                                                                                                        in:24    pkgs:1   size:21.3KB    loc:706
    

    There's actually only one package there, but it's appearing multiple times in the results (because of the external test shenanigans).

    It's probably not helpful to include these entries in the cut results because it's unlikely we're going to consider removing tests.

    In fact, these results are kinda confusing all round unless you're familiar with the way that test packages are dealt with. I know these package need to be a part of the package sets internally, but I wonder if there might be a way to hide them from user view, at least most of the time?

  • graph: collapsing modules, internal and arbitrary keywords

    graph: collapsing modules, internal and arbitrary keywords

    During output it would be nice to collapse some particular nodes into a single node.

    For example, these could be collapsed:

    1. modules
    2. */internal/*
    3. golang.org/x/tools/*
    4. everything except the project module (or multiple modules)

    This sounds like pkggraph needs collapsing and uncollapsing capabilities so that the code can be written as:

    goda graph -m .
    goda graph -collapse "*/internal/*" .
    goda graph -collapse "golang.org/x/tools/*" .
    
    # combining all together
    goda graph -m -uncollapse github.com/loov/goda -collapse "*/internal/*" .
    
Go dependency management tool experiment (deprecated)
Go dependency management tool experiment (deprecated)

Dep dep is a dependency management tool for Go. It requires Go 1.9 or newer to compile. NOTE: Dep was an official experiment to implement a package ma

Dec 23, 2022
dependency tool for go

Godep - Archived Please use dep or another tool instead. The rest of this readme is preserved for those that may still need its contents. godep helps

Dec 14, 2022
A simple dependency manager for Go (golang), inspired by Bundler.
A simple dependency manager for Go (golang), inspired by Bundler.

Goop A dependency manager for Go (golang), inspired by Bundler. It is different from other dependency managers in that it does not force you to mess w

Sep 27, 2022
Barebones dependency manager for Go.
Barebones dependency manager for Go.

Go Package Manager Go Package Manager (or gpm, for short) is a tool that helps achieve reproducible builds for Go applications by specifying the revis

Dec 14, 2022
Barebones dependency manager for Go.
Barebones dependency manager for Go.

Johnny Deps Johnny Deps is a small tool from VividCortex that provides minimalistic dependency versioning for Go repositories using Git. Its primary p

Sep 27, 2022
Spaghetti: a dependency analysis tool for Go packages
Spaghetti: a dependency analysis tool for Go packages

Spaghetti is an interactive web-based tool to help you understand the dependencies of a Go program, and to explore and evaluate various possible efforts to eliminate dependencies.

Dec 15, 2022
APKrash is an Android APK security analysis toolkit focused on comparing APKs to detect tampering and repackaging.
 APKrash is an Android APK security analysis toolkit focused on comparing APKs to detect tampering and repackaging.

APKrash APKrash is an Android APK security analysis toolkit focused on comparing APKs to detect tampering and repackaging. Features Able to analyze pu

Nov 8, 2022
A reflection based dependency injection toolkit for Go.

⚒️ dig A reflection based dependency injection toolkit for Go. Good for: Powering an application framework, e.g. Fx. Resolving the object graph during

Jan 1, 2023
CDK - Zero Dependency Container Penetration Toolkit
 CDK - Zero Dependency Container Penetration Toolkit

CDK is an open-sourced container penetration toolkit, offering stable exploitation in different slimmed containers without any OS dependency. It comes with penetration tools and many powerful PoCs/EXPs helps you to escape container and takeover K8s cluster easily.

Dec 29, 2022
Tpf2-tpnetmap-toolkit - A toolkit to create svg map images from TransportFever2 world data
Tpf2-tpnetmap-toolkit - A toolkit to create svg map images from TransportFever2 world data

tpf2-tpnetmap-toolkit TransportFever2 のワールドデータから svg のマップ画像を作成するツールキットです。 1. 導入方

Feb 17, 2022
Reduce debugging time while programming Go. Use static and stack-trace analysis to determine which func call causes the error.
Reduce debugging time while programming Go. Use static and stack-trace analysis to determine which func call causes the error.

Errlog: reduce debugging time while programming Introduction Use errlog to improve error logging and speed up debugging while you create amazing code

Nov 18, 2022
Technical Analysis Library for Golang

Techan TechAn is a technical analysis library for Go! It provides a suite of tools and frameworks to analyze financial data and make trading decisions

Dec 30, 2022
Selected Machine Learning algorithms for natural language processing and semantic analysis in Golang

Natural Language Processing Implementations of selected machine learning algorithms for natural language processing in golang. The primary focus for t

Dec 25, 2022
Selected Machine Learning algorithms for natural language processing and semantic analysis in Golang

Natural Language Processing Implementations of selected machine learning algorithms for natural language processing in golang. The primary focus for t

Dec 25, 2022
Vilicus is an open source tool that orchestrates security scans of container images(docker/oci) and centralizes all results into a database for further analysis and metrics.
Vilicus is an open source tool that orchestrates security scans of container images(docker/oci) and centralizes all results into a database for further analysis and metrics.

Vilicus Table of Contents Overview How does it work? Architecture Development Run deployment manually Usage Example of analysis Overview Vilicus is an

Dec 6, 2022
Standardized Malware Analysis Tool

S.M.A.T Standardized Malware Analysis Toolkit Capabilities Unpac.me sample submission download results check if already submitted MWDB query for confi

Oct 25, 2022
Lightweight static analysis for many languages. Find bug variants with patterns that look like source code.

Lightweight static analysis for many languages. Find bugs and enforce code standards. Semgrep is a fast, open-source, static analysis tool that finds

Jan 9, 2023
bodyclose is a static analysis tool which checks whether res.Body is correctly closed.

bodyclose is a static analysis tool which checks whether res.Body is correctly closed. Install You can get bodyclose by go get command. $ go

Jan 6, 2023