Trash - Minimalistic Go vendored code manager

Trash - Go ./vendor manager

Keeping the trash in your ./vendor dir to a minimum.

How to use

Make sure you're using go1.6 or later version.

  1. Download and extract latest release to your PATH. Alternatively, install or update current development version with go get -u github.com/rancher/trash.
  2. Copy vendor.conf file to your project and edit to your needs.
  3. Run trash.

vendor.conf (in your project root dir) specifies the revisions (git tags or commits, or branches - if you're drunk) of the libraries to be fetched, checked out and copied to ./vendor dir. For example:

github.com/rancher/trash

github.com/Sirupsen/logrus                      v0.8.7    https://github.com/imikushin/logrus.git
github.com/codegangsta/cli                      b5232bb
github.com/cloudfoundry-incubator/candiedyaml   5a459c2

Or, in YML format:

import:
- package: github.com/Sirupsen/logrus               # package name
  version: v0.8.7                                   # tag or commit
  repo:    https://github.com/imikushin/logrus.git  # (optional) git URL

- package: github.com/codegangsta/cli
  version: b5232bb2934f606f9f27a1305f1eea224e8e8b88

- package: github.com/cloudfoundry-incubator/candiedyaml
  version: 55a459c2d9da2b078f0725e5fb324823b2c71702

Run trash to populate ./vendor directory and remove unnecessary files. Run trash --keep to keep all checked out files in ./vendor dir.

Inspiration

I really liked glide, it's like a real package manager: specify what you need, run glide up and enjoy your updated libraries. But it didn't help with a couple problems I had:

  • All necessary library code should be vendored and checked into project repo (as imposed by the project policy)
  • Unnecessary code should be removed for great justice for smaller git checkouts and faster docker build

I'd been slightly reluctant to the idea of writing it, but apparently the world needed another package manager: "Come on, it's just going to be 300 (okay, it's ~600) lines of Go!" Thanks to @ibuildthecloud for the idea.

Help

For the world's convenience, trash can detect glide.yaml (and glide.yml, as well as trash.yaml) and use that instead of vendor.conf (and you can Force it to use any other file). Just in case, here's the program help:

$ trash -h
NAME:
   trash - Vendor imported packages and throw away the trash!

USAGE:
   trash [global options] command [command options] [arguments...]

VERSION:
   v0.2.7

AUTHOR(S):
   @imikushin, @ibuildthecloud

COMMANDS:
     help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --file value, -f value       Vendored packages list (default: "vendor.conf")
   --directory value, -C value  The directory in which to run, --file is relative to this (default: ".")
   --target value, -T value     The directory to store results (default: "vendor")
   --keep, -k                   Keep all downloaded vendor code (preserving .git dirs)
   --update value, -u value     specify a list of packages to be updated
   --insecure                   Pass -insecure to 'go get'
   --debug, -d                  Debug logging
   --cache value                Cache directory (default: "/Users/ivan/.trash-cache") [$TRASH_CACHE]
   --include-vendor             whether to include vendor when running trash -k
   --help, -h                   show help
   --version, -v                print the version
Comments
  • panic when exec trash

    panic when exec trash

    trash.yml:

    import:
    - package: github.com/Sirupsen/logrus               # package name
      version: v0.8.7                                   # tag or commit
      repo:    https://github.com/imikushin/logrus.git  # (optional) git URL
    
    - package: github.com/codegangsta/cli
      version: b5232bb2934f606f9f27a1305f1eea224e8e8b88
    
    - package: github.com/cloudfoundry-incubator/candiedyaml
      version: 55a459c2d9da2b078f0725e5fb324823b2c71702
    
    

    exec trash cmd, panic occured: root@0ad3b076d0bb:~/test_trash# trash INFO[0000] Trash! Reading file: 'trash.yml' INFO[0000] Preparing cache for 'github.com/Sirupsen/logrus' INFO[0009] Preparing cache for 'github.com/codegangsta/cli' INFO[0024] Preparing cache for 'github.com/cloudfoundry-incubator/candiedyaml' INFO[0031] Checking out 'github.com/Sirupsen/logrus', commit: 'v0.8.7' INFO[0031] Checking out 'github.com/codegangsta/cli', commit: 'b5232bb2934f606f9f27a1305f1eea224e8e8b88' INFO[0031] Checking out 'github.com/cloudfoundry-incubator/candiedyaml', commit: '55a459c2d9da2b078f0725e5fb324823b2c71702' INFO[0031] Copying deps... INFO[0031] Copying deps... Done INFO[0031] removing '/root/test_trash/vendor/github.com/Sirupsen/logrus/.git INFO[0031] removing '/root/test_trash/vendor/github.com/cloudfoundry-incubator/candiedyaml/.git INFO[0031] removing '/root/test_trash/vendor/github.com/codegangsta/cli/.git INFO[0031] Collecting packages in 'test_trash' panic: runtime error: invalid memory address or nil pointer dereference [signal 0xb code=0x1 addr=0x20 pc=0x40a98e] /usr/local/go/src/runtime/panic.go:481 +0x3e6 main.listPackages.func1(0xc82000e02a, 0xa, 0x0, 0x0, 0x7f8d245902a8, 0xc820105200, 0x0, 0x0) panic(0x/root/gopath/src/github.com/rancher/trash/trash.go:370 +0x4e path/fil/usr/local/go/src/path/filepath/path.go:394 +0xa50x0) main.lis/root/gopath/src/github.com/rancher/trash/trash.go:386 +0x9f main.col/root/gopath/src/github.com/rancher/trash/trash.go:394 +0x187 main.cle/root/gopath/src/github.com/rancher/trash/trash.go:515 +0x49b main.run/root/gopath/src/github.com/rancher/trash/trash.go:108 +0x626 main.mai/root/gopath/src/github.com/rancher/trash/trash.go:61 +0x21 github.c/root/gopath/src/github.com/rancher/trash/vendor/github.com/codegangsta/cli/app.go:132 +0xadb, 0x1, 0x0, 0x0) main.mai/root/gopath/src/github.com/rancher/trash/trash.go:64 +0x6e6

    go env: root@0ad3b076d0bb:~/test_trash# go env GOARCH="amd64" GOBIN="" GOEXE="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOOS="linux" GOPATH="/root/gopath" GORACE="" GOROOT="/usr/local/go" GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64" GO15VENDOREXPERIMENT="1" CC="gcc" GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0" CXX="g++" CGO_ENABLED="1"

    go version: root@0ad3b076d0bb:~/test_trash# go version go version go1.6.2 linux/amd64

  • Support exclude patterns

    Support exclude patterns

    This PR contains an implementation of exclude patterns, where the user may specify certain packages )or folders) to exclude from vendoring. For the rationale behind this please see https://github.com/rancher/trash/issues/54.

    The exclusion pattern in the plain config file format is an import path with an dash (-) preceding it, so it's unambiguous and backwards compatible with the current mode of operation. In yaml format a new exclude field is used which contains a list of paths to exclude. If an exclude pattern is present in the config file, trash will first delete those from any vendored packages and only then proceed with the removeUnusedImports step, ensuring that anything that only excluded packages depended on is also removed.


    Beside the above feature, the PR contains two bugfixes:

    Previously a trash --update overwrote any existing config file without retaining its original format (https://github.com/rancher/trash/issues/46). This had the unfortunate side effect that an original yaml config file was swapped out with a flat conf type. A proposal to fix this was done in https://github.com/rancher/trash/pull/50, but adding a new flag is a redundant operation since the code already knows what the original format was. This PR ensures that the original config file format (yaml or conf) is retained inside the Conf struct, allowing to serialize back to the same format on a dump.

    Another bug was that trash recreated a new config structure from scratch on update, overwriting the old struct before the package metadata (version, external repo) could be extracted, losing all such information. A proposal was opened in https://github.com/rancher/trash/pull/51 to fix this, but it doesn't play nice with other internal/private fields in the Conf struct. This PR offers an alternative solution by not recreating the original config struct, just its Imports field. This was all private metadata is preserved, including the importMap containing the import meta fields.

  • What is the

    What is the "unnecessary code" mentioned in the README

    One the reasons this project exists (as per the README) is that "unnecessary code" should not be in vendor dir (which is version controller). Could someone help me understand what "unnecessary code" might make its way into that dir if we used glide?

  • Error message while running trash

    Error message while running trash

    Can you tell me what I am doing wrong here:

    FATA[0000] git checkout -f --detach remote-develop failed: fatal: '--detach' cannot be used with '-b/-B/--orphan'

  • Ignore code already vendored in a nested path

    Ignore code already vendored in a nested path

    When collecting import paths to vendor, trash walked the entire folder hierarchy and pulled in everything it saw. However if a subpackage (either inside or outside the top level vendor folder) already vendored all its stuff in, that trash still found them and pulled them into the top level vendor folder too. This is kind of pointless because Go will resolve those dependencies at the closest vendor level, and won't ascend up to the top of the project, so a lot of duplicate but not actually code will be left in.

    This PR fixes this issue by ascending the project structure whenever a new import path is found. If the import can be satisfied by an already vendored library deeper in the repository, it skips pulling it into the main vendor folder.

  • Fix panics related to Walk func and also path handling in default layouts

    Fix panics related to Walk func and also path handling in default layouts

    The stock code in 0.5.0 did not work in 2 cases and would panic.

    The first was within a filepath Walker function that upon encountering an error in the walk function would not test for the error and would then panic when trying to use input parameters to get file information.

    The second is a bit mysterious. When using cleanup the go path would use multiple up directory filepath components to navigate to the top of a directory tree. If the GOPATH was set of a project in a local file system with for example /home/user/project would end up stripping important potions of the path. This is probably because it is being run in another environment that does not use a directory tree for projects that are being used with trash. I figure this is some incompatibility between the types of project structures that can be used with go.

  • no-cache flag, gopath tmp variable...

    no-cache flag, gopath tmp variable...

    ...minor format cleanup

    I have replaced GOPATH rewinding by traversing to temp original GOPATH variable, added new flag which removes cache after vendoring (can be useful for containers) and made a few code formatting and cleanup (pointed by IDE).

  • Do not attempt to preserve file permissions

    Do not attempt to preserve file permissions

    == Background

    In common configurations of NFS, it is not possible to copy file permissions. This manifests when using cp -a:

    $ cd /Volumes/Developer  # An NFS share.
    $ cp -a vendor.conf vendor.conf.bak
    cp: preserving permissions for 'vendor.conf.bak': Operation not permitted
    

    == Overview

    This diff changes the cpy method to use cp -dR instead of cp -a, which is equivalent except for it doesn't copy permissions -- see MAN CP(1):

       -a, --archive
              same as -dR --preserve=all
    

    == Testing

    With the following vendor.conf file:

    temp
    
    github.com/go-ini/ini	v1.21.1-1-g2ba15ac
    

    Previous behavior:

        temp$ $ trash -C .
    INFO[0000] Trash! Reading file: 'vendor.conf'
    INFO[0000] Using 'temp' as the project's root package (from vendor.conf)
    INFO[0000] Preparing cache for 'github.com/go-ini/ini'
    INFO[0001] Checking out 'github.com/go-ini/ini', commit: 'v1.21.1-1-g2ba15ac'
    INFO[0001] Copying deps...
    FATA[0001] `cp -a /home/vagrant/.trash-cache/src/github.com/go-ini/ini /Volumes/Developer/temp/vendor/github.com/go-ini/` failed:
    cp: preserving permissions for '/Volumes/Developer/temp/vendor/github.com/go-ini/ini/.git/refs/heads/master': Operation not permitted
    

    New behavior:

    $ trash -C .
    INFO[0000] Trash! Reading file: 'vendor.conf'
    INFO[0000] Using 'temp' as the project's root package (from vendor.conf)
    INFO[0000] Checking out 'github.com/go-ini/ini', commit: 'v1.21.1-1-g2ba15ac'
    INFO[0000] Copying deps...
    INFO[0000] Copying deps... Done
    INFO[0000] removing '/Volumes/Developer/temp/vendor/github.com/go-ini/ini/.git
    INFO[0000] Collecting packages in 'temp'
    INFO[0000] Removing unused dir: 'vendor/github.com'
    WARN[0000] Package 'github.com/go-ini/ini' has been completely removed: it's probably useless (in vendor.conf)
    
  • Detect and keep CGO dependencies from subfolders.

    Detect and keep CGO dependencies from subfolders.

    This PR fixes an issue that's haunting all the dependency managers out there (e.g. https://github.com/tools/godep/issues/422), namely that they only take into consideration Go dependencies, and not CGO ones. This results in build failures for all projects where C files are in subfolders (i.e. most of them), as the dependency manager deletes all those folders thinking they are unused.

    This fixes the issue by doing a second round of import scanning, where instead of looking for Go import statements, the code looks for CGO preambles and extracts any #include "path"-s where the path points to an existing folder. This will result in any folders references by CGO code to be kept, instead of trashed.

  • FATA[0041] It didn't work: '/home/me/code/go/src/src' does not exist or something: stat /home/me/code/go/src/src: no such file or directory

    FATA[0041] It didn't work: '/home/me/code/go/src/src' does not exist or something: stat /home/me/code/go/src/src: no such file or directory

    I got this error message when running trash:

    FATA[0041] It didn't work: '/home/me/code/go/src/src' does not exist or something: stat /home/me/code/go/src/src: no such file or directory
    

    This line was the reason.

    My project is in a path one level too deep for that line of code to work. Shouldn't getting the GOPATH be enough on that case?

    Thanks for this simple, great tool.

  • Feature Request : locating root GOPATH src directory for cleanup

    Feature Request : locating root GOPATH src directory for cleanup

    Looking at the start of the cleanup function in trash.go a relative hard coded path is used to navigate from the assumed location of the trash project to the src directory of the 3rd parties software system reusing trash.

    The intent is that this location is then used to discover where the vendor directory of the users project is and traverse the vendor directory looking for unused projects and purging them.

    The proposal is that an alternative to the hard coded path is needed for supporting diverse package structuring situations.

    Comment from Ivan 👍

    @karlmutch Go projects are supposed to have a certain directory structure. I admit we need to be more intelligent than just cd ../../../../src to get to source root. That will be addressed in a later release. Feel free to suggest a fix.

    What I'm trying to do is to guess the root package of the project trash is being run against.

    For example, rancher/os should reside in /go/src/github.com/rancher/os (if all your Go projects are using /go as GOPATH) or /home/ivan/src/rancher/os/src/github.com/rancher/os (I'm using a dedicated path as GOPATH for just RancherOS). In both cases I can tell the "root" package of the project is "github.com/rancher/os" from the directory structure of the checked out project.

    For projects that do not observe the Go Way dir structure, we probably want to use the top level package key (like in trash's own trash.yml) to get the root package.

  • Can't import trash because of the case sensitivity of github.com/Sirupsen/logrus

    Can't import trash because of the case sensitivity of github.com/Sirupsen/logrus

    When I try go get -u github.com/rancher/trash I get error:

    go get: github.com/Sirupsen/logrus@none updating to
            github.com/Sirupsen/[email protected]: parsing go.mod:
            module declares its path as: github.com/sirupsen/logrus
                    but was required as: github.com/Sirupsen/logrus
    

    This is described on the logrus site and they tell users to change. Can you please?
    Thanks,

    Bob Perper [email protected]

  • can trash also install the binary if a package has Main?

    can trash also install the binary if a package has Main?

    for package like ffjson the upstream is not using any tags, so I want to check it out at a specific version, I want something like go get github.com/pquerna/ffjson @ d49c2bc if I use this line below in vendor.conf that works almost, but one extra thing go get does is to install the binary; I want the ffjson to be installed to my $GOPATH/bin as json code generator, can trash also do install the binary?

       github.com/pquerna/ffjson         d49c2bc
    
  • Please fix the broken functionality automatically populating vendor.conf

    Please fix the broken functionality automatically populating vendor.conf

    @StrongMonkey as we discussed offline, it's very useful to run trash -u to pick up all the dependencies automatically and later adjust the versions if I need to.

    Even if this functionality is available with a different flag, that's fine too.

    The automagically populating the libraries (adding/removing) in vendor.conf is very valuable.

  • Unable to add client.go dependency in rancher-agent

    Unable to add client.go dependency in rancher-agent

    I am trying to add client.go dependency in rancher agent. I have included

    - package: k8s.io/client-go/kubernetes
      version: ee121606b0d09b2e1c467183ee46217fa85a6b67
    

    in the trash.yml and then ./vendor/github.com/docker/distribution/vendor.conf and ./vendor/github.com/docker/docker/vendor.conf

    But whenever I try update dependencies using trash update I get the following error.

    INFO[0003] Fetching latest commits from 'origin' for 'k8s.io/client-go/kubernetes' 
    FATA[0004] `git checkout -f --detach ee121606b0d09b2e1c467183ee46217fa85a6b67` failed:
    fatal: reference is not a tree: ee121606b0d09b2e1c467183ee46217fa85a6b67
    

    When I try to build agent I get the following:

    common.go:11:2: cannot find package "k8s.io/client-go/kubernetes" in any of:
    	/go/src/github.com/rancher/agent/vendor/k8s.io/client-go/kubernetes (vendor tree)
    	/usr/local/go/src/k8s.io/client-go/kubernetes (from $GOROOT)
    	/go/src/k8s.io/client-go/kubernetes (from $GOPATH)
     ...
    
  • trash --keep is not downloading vendor files

    trash --keep is not downloading vendor files

    Version info:

    VERSION:
       v0.3.0-dev
    

    I am running, trash --keep, nothing happens except for the following log messages:

    DEBU[0000] dir: '/home/greenpau/go/src/github.com/greenpau/testrepo'
    INFO[0000] Trash! Reading file: 'vendor.conf'
    

    If I run trash --update and then run trash --keep, it does download the packages, but it also updates Git version references in vendor.conf.

    The desired outcome is to run trash --keep to download dependencies to .vendor directory. The versioning of the dependencies should be based on the versions in vendor.conf.

Related tags
Sloc, Cloc and Code: scc is a very fast accurate code counter with complexity calculations and COCOMO estimates written in pure Go
Sloc, Cloc and Code: scc is a very fast accurate code counter with complexity calculations and COCOMO estimates written in pure Go

Sloc Cloc and Code (scc) A tool similar to cloc, sloccount and tokei. For counting physical the lines of code, blank lines, comment lines, and physica

Jan 8, 2023
AI-powered code snippet generator using ChatGPT. Generate code in various languages based on natural language descriptions!

SnipForge - AI-Powered Code Snippet Generator SnipForge is a powerful command-line interface (CLI) tool that utilizes OpenAI's GPT technology to gener

May 5, 2023
:zap: boilerplate template manager that generates files or directories from template repositories
:zap: boilerplate template manager that generates files or directories from template repositories

Boilr Are you doing the same steps over and over again every time you start a new programming project? Boilr is here to help you create projects from

Jan 6, 2023
The slightly more awesome standard unix password manager for teams
The slightly more awesome standard unix password manager for teams

gopass Introduction gopass is a password manager for the command line written in Go. It supports all major operating systems (Linux, MacOS, BSD) as we

Jan 4, 2023
Command Line Alias Manager and Plugin System - Written in Golang
Command Line Alias Manager and Plugin System - Written in Golang

aly - Command Line Alias Manager and Packager Aly offers the simplest way to manage, share, and obtain command line aliases! Warning: This project is

Jun 16, 2022
The Todo List / Task Manager for Geeks in command line
The Todo List / Task Manager for Geeks in command line

The CLI To-Do List / Task Manager for Geeks ??‍?? Developer / DevOps / Sysadmin? A command line hero? ?? Live with the dark terminal? ?? Think in Mark

Dec 15, 2022
Secure, private and feature-rich CLI password manager
Secure, private and feature-rich CLI password manager

Kure Kure is a free and open-source password manager for the command-line. This project aims to offer the most secure and private way of operating wit

Nov 17, 2022
file manager on terminal written in Go
file manager on terminal written in Go

ff ff is file manager written in Go. Features preview file/directory copy/paste file make a new file/directory rename a file/directory edit file with

Nov 18, 2022
GodSpeed is a robust and intuitive manager for reverse shells.
GodSpeed is a robust and intuitive manager for reverse shells.

GodSpeed is a robust and intuitive manager for reverse shells. It supports tab-completion, verbose listing of connected hosts and easy interaction with selected shells by passing their corresponding ID.

Dec 12, 2022
A terminal based file manager
A terminal based file manager

Keep those files organized About The Project A terminal based file manager Built With Go bubbletea bubbles lipgloss Installation go install github.com

Dec 27, 2022
Teller - the open-source universal secret manager for developers
 Teller - the open-source universal secret manager for developers

A secrets management tool for developers built in Go - never leave your command line for secrets.

Jan 8, 2023
A multiple reverse shell sessions/clients manager via terminal written in go

A multiple reverse shell sessions/clients manager via terminal written in go

Nov 9, 2022
Moldy CLI the best project starter and manager of the world
Moldy CLI the best project starter and manager of the world

You don't know how to start your project ... you want to help other people know your tool or language. Use Moldy! the best helper to start your project

Oct 17, 2022
Bucket-ssh. A fuzzy ssh manager for managing and categorizing ssh connections.
Bucket-ssh. A fuzzy ssh manager for managing and categorizing ssh connections.

Bssh is an ssh bucket for categorizing and automating ssh connections. Also, with parallel command execution and connection checks(pings) over categories (namespaces).

Oct 25, 2022
Declarative CLI Version manager. Support Lazy Install and Sharable configuration mechanism named Registry. Switch versions seamlessly

aqua Declarative CLI Version manager. Support Lazy Install and Sharable configuration mechanism named Registry. Switch versions seamlessly. Index Slid

Dec 29, 2022
CLI todo manager for iCalendar files.

tdx tdx is a todo manager for iCalendar files. It is designed to work with Vdirsyncer and expects to operate in its storage format. iCalendar-compatib

Dec 23, 2022
A terminal-based download manager written in Go!

goload-manager About A terminal-based download manager written in Go! This project uses grab for downloading and tcell for the terminal interface. Hea

Oct 7, 2021
A CLI password manager written in Go

vlt A CLI password vault written in Go vlt is a command line tool for storing passwords in encrypted files. Files are stored on a per-user basis in $H

Oct 23, 2021
Grit is an experimental personal task manager that represents tasks as nodes of a multitree
Grit is an experimental personal task manager that represents tasks as nodes of a multitree

Grit is an experimental personal task manager that represents tasks as nodes of a multitree, a class of directed acyclic graphs. The structure en

Jan 2, 2023