Gohack: mutable checkouts of Go module dependencies

Gohack: mutable checkouts of Go module dependencies

The new Go module system is awesome. It ensures repeatable, deterministic builds of Go code. External module code is cached locally in a read-only directory, which is great for reproducibility. But if you're used to the global mutable namespace that is $GOPATH, there's an obvious question: what if I'm hacking on my program and I want to change one of those external modules?

You might want to put a sneaky log.Printf statement to find out how some internal data structure works, or perhaps try out a bug fix to see if it solves your latest problem. But since all those external modules are in read-only directories, it's hard to change them. And you really don't want to change them anyway, because that will break the integrity checking that the Go tool does when building.

Luckily the modules system provides a way around this: you can add a replace statement to the go.mod file which substitutes the contents of a directory holding a module for the readonly cached copy. You can of course do this manually, but gohack aims to make this process pain-free.

Install gohack with

go get github.com/rogpeppe/gohack

or use gobin:

gobin github.com/rogpeppe/gohack

For quick edits to a module (without version control information)

If the module to edit is example.com/foo/bar, run:

gohack get example.com/foo/bar

This will make a copy of the module into $HOME/gohack/example.com/foo/bar and add replace directives to the local go.mod file:

replace example.com/foo/bar => /home/rog/gohack/example.com/foo/bar

Note: This copy will not include version control system information so it is best for quick edits that aren't intended to land back into version control.

To edit the module with full version control

Run:

gohack get -vcs example.com/foo/bar

This will clone the module's repository to $HOME/gohack/example.com/foo/bar, check out the correct version of the source code there and add the replace directive into the local go.mod file.

Undoing replacements

Once you are done hacking and wish to revert to the immutable version, you can remove the replace statement with:

gohack undo example.com/foo/bar

or you can remove all gohack replace statements with:

gohack undo

Note that undoing a replace does not remove the external module's directory - that stays around so your changes are not lost. For example, you might wish to turn that bug fix into an upstream PR.

If you run gohack on a module that already has a directory, gohack will try to check out the current version without recreating the repository, but only if the directory is clean - it won't overwrite your changes until you've committed or undone them.

Comments
  • ability to place gohack files inside module directory instead of $HOME

    ability to place gohack files inside module directory instead of $HOME

    I haven't tried gohack yet, but I am curious about this scenario which I have encountered when doing a similar thing manually:

    I have two separate projects (let's call them myProjectA and myProjectB), each of which happen to use the github.com/jmoiron/sqlx package.

    For project myProjectA, I discover a need to modify function f of sqlx, so I use gohack to get a mutable copy of the repo, and make my changes.

    For project myProjectB, I want to use the original unmodified version of sqlx, but I want to place a log.Printf(...) statement inside function g of sqlx. If I were to use gohack to replace sqlx in this project, my understanding is that project myProjectB would begin using the same copy of sqlx as project myProjectA.

    Is my reasoning correct?

    If so, a way to avoid this seems to be to place gohack'd packages inside, say, myProjectA/.gohack/github.com/jmoiron/sqlx. This may also resolve #5?

    I suppose the main issues with this are:

    • Having a git repo within a git repo - however, the .gohack directory would most likely be included in .gitignore
    • More difficult to use the same gohack'd package across multiple modules
  • cannot get module info

    cannot get module info

    I am getting error

    $ gohack get -vcs gopkg.in/danilopolani
    cannot get module info: go list -m: can't compute 'all' using the vendor directory
    	(Use -mod=mod or -mod=readonly to bypass.)
    
  • do not use go list to find go.mod file

    do not use go list to find go.mod file

    The go list command fails if the go.mod file isn't well formed (like for example one of the replace directives refers to a non-existent target). This can be a problem, so find the go.mod file by looking directly for it instead.

  • cannot update VCS dir for github.com/labstack/echo: error: pathspec 'v3.2.1+incompatible' did not match any file(s) known to git.

    cannot update VCS dir for github.com/labstack/echo: error: pathspec 'v3.2.1+incompatible' did not match any file(s) known to git.

    when run gohack github.com/labstack/echo

    it show this error message: creating github.com/labstack/[email protected]+incompatible cannot update VCS dir for github.com/labstack/echo: error: pathspec 'v3.2.1+incompatible' did not match any file(s) known to git. all modules failed; not replacing anything

  • multiple subcommands

    multiple subcommands

    As the amount of functionality grows, it seems like we should consider having subcommands.

    A possible set of commands:

     gohack get [-vcs] [-u] [-f] [module...]
    

    Get gets the modules at the current version and adds replace statements to the go.mod file if they're not already replaced. If the -u flag is provided, the source code will also be updated to the current version if it's clean. If the -f flag is provided with -u, the source code will be updated even if it's not clean. If the -vcs flag is provided, it also checks out VCS information for the modules. If the modules were already gohacked in non-VCS mode, gohack switches them to VCS mode, preserving any changes made (this might result in the directory moving).

    With no module arguments and the -u flag, it will try to update all currently gohacked modules.

    gohack diff module
    

    Diff prints (in git style) changes that have been made to the module since it was checked out.

    gohack rm [-f] module...
    

    Rm removes the gohack directory if it is clean and then runs gohack undo. If the -f flag is provided, the directory is removed even if it's not clean.

     gohack undo [module...]
    

    Undo removes the replace statements for the modules. If no modules are provided, it will undo all gohack replace statements. The gohack module directories are unaffected.

    gohack dir [-vcs] [module...]
    

    Dir prints the gohack module directory names for the given modules. If no modules are given, all the currently gohacked module directories are printed. If the -vcs flag is provided, the directory to be used in VCS mode is printed. Unlike the other subcommands, the modules don't need to be referenced by the current module.

  • Ability to hack on an already replaced module

    Ability to hack on an already replaced module

    I think this makes sense; based on an actual scenario.

    A module M I was working on had a dependency on D1. But I'd already forked D1 to D1'. Hence my go.mod already had a replace from D1 => D1'.

    So my desire to hack on D1 translates to needing to hack on D1'. And hence gohack should be following replace directives to work out what should ultimately be hacked on. Instead I got:

    "github.com/shurcooL/vfsgen" is already replaced; will not override replace statement in go.mod
    all modules failed; not replacing anything
    error: [
            {/home/myitcv/gostuff/src/gopkg.in/errgo.v2/fmt/errors/alias.go:15: all modules failed; not replacing anything}
    ]
    
  • Don't pollute $HOME by default

    Don't pollute $HOME by default

    I know the purpose of ~/gohack, but I'm not a big fan of tools that add stuff to my home directory by default.

    Perhaps we could use $GOPATH/hack or something else instead. Don't have a clear answer to this one.

  • Idea: flag for

    Idea: flag for "fork" remote

    So the workflow I just used was the following:

    • use gohack to pull down depedenency I want to add log statements too or changes
    • cd int ~/gohack/github.com/ldelossa/repo
    • added a remote called "fork" which points to my fork of github.com/ldelossa/repo
    • hacked on code till I got what I wanted, pushed branch to my fork
    • open PR of fork with original repo.

    Obviously the fork will need to be setup everytime we run GoHack. Curious if you think it's a good idea to add a "remote" or "fork" cli flag to gohack which setups a remote fork to push your changes too in an automated fashion. If so I'd like to take a stab at this PR.

  • main: fix definition and implementation of relative GOHACK

    main: fix definition and implementation of relative GOHACK

    rogpeppe has a far sharper brain than me and pointed out a major issue with the previous implementation: that we hadn't actually implemented a relative value of GOHACK relative to the main module.

  • gohack does not allow relative directories in GOHACK env var.

    gohack does not allow relative directories in GOHACK env var.

    We want to be able to set GOHACK to a relative dir, say ./. But this is not possible at the moment as the initial ./ is removed in the replace directive, creating a broken replacement line for go:

    replacement module without version must be directory path (rooted or starting with ./ or ../)
    

    We can work around the problem by manually prefixing the replacement with ./.

    We like the relative path use-case, because it allows enhanced interop in WSL, when the path is on a shared Windows mount (/mnt/c/...), Windows go tools can interpret the same (relative) path as the WSL go tools. With absolute paths, Windows tools obviously cannot interpret the /mnt/c/... mount.

    This is somewhat related to #27.

  • provide way to print directory without doing anything else

    provide way to print directory without doing anything else

    It would be useful to be able to print the gohack directory for a module without actually making any changes.

    Perhaps:

    gohack -p modulepath
    

    would print the directory for modulepath. With no arguments, -p could print the directories for all the modules in the current module that have gohack directories.

  • supporting workspaces?

    supporting workspaces?

    Would it be reasonable for gohack to support adding a module to a workspace (if a go.work file is present) rather than adding a replace to the go.mod file?

  • Imply module name from $PWD

    Imply module name from $PWD

    -*- mode: compilation; default-directory: "~/go/pkg/mod/github.com/stapelberg/[email protected]/" -*-
    Compilation started at Fri May 29 21:53:41
    
    goversion -m $(which gohack) && gohack get
    /home/michael/go/bin/gohack go1.14
    	path  github.com/rogpeppe/gohack
    	mod   github.com/rogpeppe/gohack       v1.0.2
    	dep   github.com/rogpeppe/go-internal  v1.0.0
    	dep   golang.org/x/tools               v0.0.0-20180917221912-90fa682c2a6e
    	dep   gopkg.in/errgo.v2                v2.1.0
    cannot determine main module: go list -m: not using modules
    
    Compilation exited abnormally with code 1 at Fri May 29 21:53:41
    

    I have to use gohack get github.com/stapelberg/glog even though it’s obvious from my working directory which package I mean.

    Would a PR for this be accepted?

  • gohack undo leaves trailing newlines

    gohack undo leaves trailing newlines

    This is how we can reproduce the issue:

    $ git clone https://github.com/rogpeppe/go-internal.git
    $ gohack get gopkg.in/errgo.v2
    $ gohack undo
    $ git status
    On branch master
    Your branch is up to date with 'origin/master'.
    
    Changes not staged for commit:
      (use "git add <file>..." to update what will be committed)
      (use "git restore <file>..." to discard changes in working directory)
            modified:   go.mod
    
    no changes added to commit (use "git add" and/or "git commit -a")
    

    git diff produces:

    diff --git a/go.mod b/go.mod
    index 1c11744..e980eb2 100644
    --- a/go.mod
    +++ b/go.mod
    @@ -3,3 +3,5 @@ module github.com/rogpeppe/go-internal
     go 1.11
    
     require gopkg.in/errgo.v2 v2.1.0
    +
    +
    

    What I expected is that it there should be no diff, i.e.

    $ git status
    On branch master
    Your branch is up to date with 'origin/master'.
    
    nothing to commit, working tree clean
    

    Looking at the Go's source code https://github.com/golang/go/blob/26154f31ad6c801d8bad5ef58df1e9263c6beec7/src/cmd/go/internal/modcmd/edit.go#L209

    maybe we could include modf.Cleanup() before modf.Format() in https://github.com/rogpeppe/gohack/blob/03d2ff3646b7ffc380e059413e4302f6cbdeb09b/gomodcmd.go#L91

    Created PR #69 for this. :smile:

  • gohack redo

    gohack redo

    When doing undo in the folder <somepath>, gohack should save what was undone to the file $HOME/gohack/<somepath> or to the file .gohack-redo. Command gohack redo should undo the undoing, so basically turn back what was removed (and delete redo file).

    Use case in my mind is the following:

    1. I'm writing the pre-commit hook, which is calling gohack undo
    2. I'm writing post-commit hook, which is calling gohack redo
    3. (optionally) I'm including .gohack-redo file in the .gitignore to not accidentally commit it
    4. I'm working as usual, being more or less sure, that replace directives would not be commited

    That would be particulary convenient for multi-repo applications, which has a shared codebase in a separate repos, because allows working on the cross-repo feature (/service1 /service2 and /lib1 /lib2 folders cloned under one root), while keeping CI builds intact (which only have /service1 cloned).

Fetch license information for all direct and indirect dependencies of your Golang project
Fetch license information for all direct and indirect dependencies of your Golang project

gocomply beta Give open source Golang developers the credit they deserve, follow your legal obligations, and save time with gocomply. This tiny little

Nov 1, 2022
depaware makes you aware of your Go dependencies

depaware depaware makes you aware of your Go dependencies. It generates a list of your dependencies which you check in to your repo: https://github.co

Dec 20, 2022
Go-htutil - Go HTTP utilities, with no dependencies

snai.pe/go-htutil go get snai.pe/go-htutil Go HTTP utilities with no dependenci

Jan 26, 2022
Checks if there are any updates for imports in your module.

Go Up goup checks if there are any updates for imports in your module. It parses go.mod files to get dependencies with their version, uses go-git to r

Jul 7, 2022
dagger is a fast, concurrency safe, mutable, in-memory directed graph library with zero dependencies
dagger is a fast, concurrency safe, mutable, in-memory directed graph library with zero dependencies

dagger is a blazing fast, concurrency safe, mutable, in-memory directed graph implementation with zero dependencies

Dec 19, 2022
A lightweight Vault client module written in Go, with no dependencies, that is intuitive and user-friendly

libvault A lightweight Hashicorp Vault client written in Go, with no dependencies. It aims to provide an intuitive, simple API that is easy to use. Ju

Sep 18, 2022
CLI client (and Golang module) for deps.dev API. Free access to dependencies, licenses, advisories, and other critical health and security signals for open source package versions.
CLI client (and Golang module) for deps.dev API. Free access to dependencies, licenses, advisories, and other critical health and security signals for open source package versions.

depsdev CLI client (and Golang module) for deps.dev API. Free access to dependencies, licenses, advisories, and other critical health and security sig

May 11, 2023
PHP functions implementation to Golang. This package is for the Go beginners who have developed PHP code before. You can use PHP like functions in your app, module etc. when you add this module to your project.

PHP Functions for Golang - phpfuncs PHP functions implementation to Golang. This package is for the Go beginners who have developed PHP code before. Y

Dec 30, 2022
Idiomatic Go input parsing with subcommands, positional values, and flags at any position. No required project or package layout and no external dependencies.
Idiomatic Go input parsing with subcommands, positional values, and flags at any position. No required project or package layout and no external dependencies.

Sensible and fast command-line flag parsing with excellent support for subcommands and positional values. Flags can be at any position. Flaggy has no

Jan 1, 2023
Go package to make lightweight ASCII line graph ╭┈╯ in command line apps with no other dependencies.
Go package to make lightweight ASCII line graph ╭┈╯ in command line apps with no other dependencies.

asciigraph Go package to make lightweight ASCII line graphs ╭┈╯. Installation go get github.com/guptarohit/asciigraph Usage Basic graph package main

Jan 1, 2023
Query, update and convert data structures from the command line. Comparable to jq/yq but supports JSON, TOML, YAML, XML and CSV with zero runtime dependencies.
Query, update and convert data structures from the command line. Comparable to jq/yq but supports JSON, TOML, YAML, XML and CSV with zero runtime dependencies.

dasel Dasel (short for data-selector) allows you to query and modify data structures using selector strings. Comparable to jq / yq, but supports JSON,

Jan 2, 2023
Dec 27, 2022
BPG decoder for Go (Zero Dependencies).

Go语言QQ群: 102319854, 1055927514 凹语言(凹读音“Wa”)(The Wa Programming Language): https://github.com/wa-lang/wa BPG for Go BPG is defined at: http://bellard.o

Sep 7, 2020
Go bindings for OpenCV1.1 (Dev/Zero Dependencies).
Go bindings for OpenCV1.1 (Dev/Zero Dependencies).

Go语言QQ群: 102319854, 1055927514 凹语言(凹读音“Wa”)(The Wa Programming Language): https://github.com/wa-lang/wa Go bindings for OpenCV1.1 PkgDoc: http://godoc

Dec 6, 2022
Rich TIFF/BigTIFF/GeoTIFF decoder/encoder for Go (Pure Go/Zero Dependencies)

Go语言QQ群: 102319854, 1055927514 凹语言(凹读音“Wa”)(The Wa Programming Language): https://github.com/wa-lang/wa TIFF for Go Features: Support BigTiff Support

Nov 16, 2022
WebP decoder and encoder for Go (Zero Dependencies).
WebP decoder and encoder for Go (Zero Dependencies).

Go语言QQ群: 102319854, 1055927514 凹语言(凹读音“Wa”)(The Wa Programming Language): https://github.com/wa-lang/wa webp ██╗ ██╗███████╗██████╗ ██████╗ ██║

Dec 28, 2022
Simply way to control goroutines execution order based on dependencies
Simply way to control goroutines execution order based on dependencies

Goflow Goflow is a simply package to control goroutines execution order based on dependencies. It works similar to async.auto from node.js async packa

Dec 8, 2022
A well tested and comprehensive Golang statistics library package with no dependencies.

Stats - Golang Statistics Package A well tested and comprehensive Golang statistics library / package / module with no dependencies. If you have any s

Dec 27, 2022
Vendor Go dependencies

nut Gophers love nuts. nut is a tool that allows Go projects to declare dependencies, download dependencies, rewrite import paths and ensure that depe

Sep 27, 2022