🏃‍♂️ A new way to execute commands and manipulate command output in Go

🏃‍♂️ run

Go Reference Tests Coverage

A new way to execute commands in Go. golangscript

Example usage

package main

import (
  "bytes"
  "context"
  "fmt"
  "io"
  "log"
  "os"

  "github.com/sourcegraph/run"
)

func main() {
  ctx := context.Background()

  // Easily stream all output back to standard out
  err := run.Cmd(ctx, "echo", "hello world").Run().Stream(os.Stdout)
  if err != nil {
    log.Fatal(err.Error())
  }

  // Or collect, map, and modify output, then collect string lines from it
  lines, err := run.Cmd(ctx, "ls").Run().
    Map(func(ctx context.Context, line []byte, dst io.Writer) (int, error) {
      if !bytes.HasSuffix(line, []byte(".go")) {
        return 0, nil
      }
      return dst.Write(bytes.TrimSuffix(line, []byte(".go")))
    }).
    Lines()
  if err != nil {
    log.Fatal(err.Error())
  }
  for i, l := range lines {
    fmt.Printf("line %d: %q\n", i, l)
  }

  // Errors include standard error by default, so we can just stream stdout.
  err = run.Cmd(ctx, "ls", "foobar").Run().StdOut().Stream(os.Stdout)
  if err != nil {
    println(err.Error()) // exit status 1: ls: foobar: No such file or directory
  }

  // Generate data from a file, replacing tabs with spaces for Markdown purposes
  var exampleData bytes.Buffer
  exampleData.Write([]byte(exampleStart + "\n\n```go\n"))
  if err = run.Cmd(ctx, "cat", "cmd/example/main.go").Run().
    Map(func(ctx context.Context, line []byte, dst io.Writer) (int, error) {
      return dst.Write(bytes.ReplaceAll(line, []byte("\t"), []byte("  ")))
    }).
    Stream(&exampleData); err != nil {
    log.Fatal(err)
  }
  exampleData.Write([]byte("```\n\n" + exampleEnd))

  // Render new README file
  var readmeData bytes.Buffer
  if err = run.Cmd(ctx, "cat", "README.md").Run().Stream(&readmeData); err != nil {
    log.Fatal(err)
  }
  replaced := exampleBlockRegexp.ReplaceAll(readmeData.Bytes(), exampleData.Bytes())

  // Pipe data to command
  err = run.Cmd(ctx, "cp /dev/stdin README.md").Input(bytes.NewReader(replaced)).Run().Wait()
  if err != nil {
    log.Fatal(err)
  }
}
Owner
Sourcegraph
Code search and navigation for teams (self-hosted, OSS)
Sourcegraph
Comments
  • input/output cannot handle very large content

    input/output cannot handle very large content

    the example contains sensitive-ish information, hence it's in a private repo

    https://github.com/sourcegraph/devops-gist/tree/main/michaellzc/2022-05-13-01

  • output: rework LineFilter API as LineMap

    output: rework LineFilter API as LineMap

    Reworks the LineFilter API to include context and be more flexible to accommodate chaining patterns based on my thoughts in https://github.com/sourcegraph/run/issues/14 and this thread, for example:

    func main() {
    	ctx, cancel := context.WithCancel(context.Background())
    
    	// Demonstrate that output streams live!
    	cmd := run.Bash(ctx, `for i in {1..10}; do echo -n "This is a test in loop $i "; date ; sleep 1; done`)
    	if err := cmd.Run().
    		Map(func(ctx context.Context, line []byte, dst io.Writer) (int, error) {
    			if bytes.Contains(line, []byte("loop 3")) {
    				defer cancel() // Interrupt parent context here
    
    				written, err := run.Cmd(ctx, "echo", "Loop 3 detected, running a subcommand!").
    					Run().
    					WriteTo(dst)
    				return int(written), err
    			}
    
    			return dst.Write(line)
    		}).
    		Stream(os.Stdout); err != nil {
    		log.Fatal(err)
    	}
    }
    

    Definitely starting to get a bit complicated code-wise though. Also, to enable chaining patterns on LineFilter we can't flush the output until all LineFilters are executed, which can be quite awkward for streaming long-running commands. That said if those subcommand want streaming output, they should stream to os.Stdout instead, so maybe this all makes sense 😁

  • quoting when building commands

    quoting when building commands

    Right now quoting must be done manually with run.Cmd, e.g. https://sourcegraph.com/github.com/sourcegraph/run@main/-/blob/exit_code_test.go?L13:35

    Maybe this is okay, maybe we want something a bit more succint? We could also perhaps alias shell.Quote so one can use run.Quote instead

  • add BashWithOpts command for strict bash exec

    add BashWithOpts command for strict bash exec

    In https://github.com/sourcegraph/run/issues/44 a command seems to not return an error from a command that is executed. Upon further investigation it is because the command is executed with bash -c, which means bash default options which means that if the last command of a pipe succeeds, the entire pipe statement is considered to have succeeded regardless if earlier statements have failed. A common "fix" for this is to enable pipefail and errexit options in bash.

    This PR adds easier way for users to execute some bash scripts with "strict" bash and possibly avoid unexpected errors.

    Closes #44

  • Cannot pass strings with spaces to Cmd

    Cannot pass strings with spaces to Cmd

    For example,

    run.Cmd(ctx, "gcloud", "compute", "disks", "snapshot",
    		conf.CurrentInstanceDisk(), "--zone", conf.CurrentInstanceZone(),
    		"--description", "mg cli driven backup",
    		"--snapshot-names", snapshotName,
    	).Run().StdOut().Stream(os.Stdout)
    

    This will pass mg cli driven backup as separate commands.

    How should we tell Cmd to not split certain strings?

  • output: add String() (string, error)

    output: add String() (string, error)

    Closes https://github.com/sourcegraph/run/issues/28 by adding a (more efficient) alias for the common pattern of getting Lines() and joining it back into a single string.

  • command: lift StdOut, StdErr into command, lift aggregator into output

    command: lift StdOut, StdErr into command, lift aggregator into output

    There doesn't seem to be a use case for changing StdOut, StdErr on-the-fly with Output, so we lift that configuration into Command, which greatly simplifies the initialisation of Output and reduces the amount of buffers we need to keep around. It also means that Output is almost purely about consuming output, which means aggregator is no longer needed, which also makes the story a lot simpler.

    API-wise, the change looks like:

    -  err = run.Cmd(ctx, "ls", "foobar").Run().StdOut().Stream(os.Stdout)
    +  err = run.Cmd(ctx, "ls", "foobar").StdOut().Run().Stream(os.Stdout)
    

    Closes https://github.com/sourcegraph/run/issues/27

    In the future, this could also improve the possibility of https://github.com/sourcegraph/run/issues/5 by not having the full command be created immediately - instead we can export specifically the customizable components

  • script paradigm

    script paradigm

    Second thoughts on scripting paradigms - in many places in sg we build entire series of commands to run. You can build []*Command, but that means you wont be able to do something like piping from one command to another.

    An idea I had was:

    type Runner interface {
       Run() Output
       Input(io.Reader) Runner
       pipeToNext() bool
    }
    
    func Script(runners ...Runner) Runner
    
    run.Script(
       run.Pipe(run.Cmd(ctx, "..."), func(o run.Output) runOutput { ... }) // run next command with Output as Input
       run.Cmd(ctx, "..."),
    )
    

    Not a huge fan of the ergonomics of the above though. Input(io.Reader) Runner also makes *Command complicated

    could be related to https://github.com/sourcegraph/run/issues/5

  • jq support

    jq support

    https://github.com/sourcegraph/run/blob/main/output.go#L34

    Another thought: maybe we can just implement this as LineFilter? e.g.

    run.Cmd(...).Run().Filter(run.JQ("...")).Lines()
    

    cc @jhchabran

  • sourcegraph/run: investigate errored command not being thrown as an error

    sourcegraph/run: investigate errored command not being thrown as an error

    See scratch log entry for context.

    Starting point is to recreate what happened:

    • run.Cmd().Run() for a gcloud command that fails due to missing permissions (maybe worthwhile to use the exact command from this issue)
    • See if gcloud returns an appropriate exit code when an error occurs, and why this is not returned as an error by run
  • instrumentation: add run.LogCommands, run.TraceCommands

    instrumentation: add run.LogCommands, run.TraceCommands

    Proposes two ways to instrument an application that executes a lot of commands you might want to debug:

    1. run.TraceCommands toggles OpenTelemetry tracing of commands executed within a context
    2. run.LogCommands toggles logging of executed commands within a context

    See instrumentation_test.go for sample usages.

    In https://github.com/sourcegraph/controller, tracing could become used as audit log type things, while logging could be useful for debugging in general.

    Open to ideas here :)

  • testing story (e.g. mock shell environments)

    testing story (e.g. mock shell environments)

    Some ideas:

    1. encourage scripts to accept Output instances, provide outputtest package for building simple mocks
    2. provide a "command factory" analogous to Buildkite's Shell, e.g. shell.Cmd() - this factory could be an interface that facilitates mocking. Alternatively, it could be shell.Run(Runner), where *Command implements Runner (now that I think of it, I like the latter):
      out := shell.Run(run.Cmd(ctx, "foobar")) // shell can mock execution and output, or propagate shared env
      

    cc @jhchabran

Executor - Wrapper for exec.Command for simple using and multi commands executing

executor Examples package main import ( "fmt" "github.com/solar-jsoc/execut

Feb 12, 2022
A simple Cron library for go that can execute closures or functions at varying intervals, from once a second to once a year on a specific date and time. Primarily for web applications and long running daemons.

Cron.go This is a simple library to handle scheduled tasks. Tasks can be run in a minimum delay of once a second--for which Cron isn't actually design

Dec 17, 2022
Colorize (highlight) `go build` command output
Colorize (highlight) `go build` command output

colorgo colorgo is a wrapper to go command that colorizes output from go build and go test. Installation go get -u github.com/songgao/colorgo Usage c

Dec 18, 2022
Run The World. Command aggregator output. Define many services watch them in one place.

Run The World. Command aggregator output. Define many services watch them in one place.

Feb 2, 2022
K3ai Executor is the runner pod to execute the "one-click" pipelines
K3ai Executor is the runner pod to execute the

Welcome to K3ai Project K3ai is a lightweight tool to get an AI Infrastructure Stack up in minutes not days. NOTE on the K3ai origins Original K3ai Pr

Nov 11, 2021
This package provides simple graph to execute functions in a group

Introduction This package provides simple graph to execute functions in a group.

Jul 22, 2022
Ghdl - A much more convenient way to download GitHub release binaries on the command line, works on Win & Unix-like systems

ghdl Memorize ghdl as github download ghdl is a fast and simple program (and als

Oct 12, 2022
The new home of the CUE language! Validate and define text-based and dynamic configuration

The CUE Data Constraint Language Configure, Unify, Execute CUE is an open source data constraint language which aims to simplify tasks involving defin

Dec 31, 2022
Day-1 is apart of my 6 days of Christmas challenge where i write in two new languages everyday, and make something weird out of it.

Day-1 is apart of my 6 days of Christmas challenge where i write in two new languages everyday, and make something weird out of it. today was a HTTP server written with PostGreSQL using Golang, R, and shell script read more

Dec 21, 2021
go.pipeline is a utility library that imitates unix pipeline. It simplifies chaining unix commands (and other stuff) in Go.

go.pipeline go.pipeline is a utility library that imitates unix pipeline. It simplifies chaining unix commands (and other stuff) in Go. Installation g

May 8, 2022
A cli for fetching the status and full output of CircleCI jobs.

CCI A cli for fetching the status and full output of CircleCI jobs. Install go install github.com/tmessi/cci/cci@latest Usage cci is designed to have

Oct 29, 2021
Parse a shell script and output all export declarations in an easy to read format

Find Exports Parse a shell script and output all export declarations in an easy to read format. Usage Example $ findexports ~/.bashrc PATH=$PATH:/usr/

Jan 13, 2022
GoThanks automatically stars Go's official repository and your go.mod github dependencies, providing a simple way to say thanks to the maintainers of the modules you use and the contributors of Go itself.
GoThanks automatically stars Go's official repository and your go.mod github dependencies, providing a simple way  to say thanks to the maintainers of the modules you use and the contributors of Go itself.

Give thanks (in the form of a GitHub ★) to your fellow Go modules maintainers. About GoThanks performs the following operations Sends a star to Go's r

Dec 24, 2022
Go library for Common Lisp format style output

format This library has the goal to bring the Common Lisp format directive to Go. This is work-in-progress, see the summary implementation table below

Jul 7, 2020
rsync wrapper (or output parser) that pushes metrics to prometheus

rsync-prom An rsync wrapper (or output parser) that pushes metrics to prometheus. This allows you to then build dashboards and alerting for your rsync

Dec 11, 2022
Generic mapStringInterface tool for extracting of data for CSV output

Generic mapStringInterface tool for extracting of data for CSV output

Nov 2, 2021
vtysock is a vtysh replacement that directly sends commands to the vty sockets of the daemons

vtysock vtysock is a vtysh replacement that directly sends commands to the vty sockets of the daemons. By skipping the parsing and validation checks d

Dec 18, 2022
A command-line tool that triggers command when the input (doesn't) comes from STDIN in an interval.

conk A command-line tool that triggers command when the input (doesn't) comes from STDIN in an interval. Usage $ conk -h A command-line tool that trig

Apr 29, 2022
An easy way to add useful startup banners into your Go applications
An easy way to add useful startup banners into your Go applications

Try browsing the code on Sourcegraph! Banner Add beautiful banners into your Go applications Table of Contents Motivation Usage API Command line flags

Jan 1, 2023