Enable your Golang applications to self update with S3

s3update

Enable your Golang applications to self update with S3. Requires Go 1.8+

This package enables our internal tools to be updated when new commits to their master branch are pushed to Github.

Latest binaries are hosted on S3 under a specific bucket along its current version. When ran locally, the binary will fetch the version and if its local version is older than the remote one, the new binary will get fetched and will exit, stating to the user that it got updated and need to be ran again.

In our case, we're only shipping Linux and Darwin, targeting amd64 platform.

Bucket will have the following structure:

mybucket/
  mytool/
	VERSION
	mytool-linux-amd64
	mytool-darwin-amd64

Usage

Updates are easier to deal with when done through a continuous integration platform. We're using CircleCI but the following excerpt can easily be adapted to whichever solution being used.

CircleCI

xgo is being used to easily cross-compile code.

Adding the following at the end of the build script will push the binaries and its version to S3.

xgo --targets="linux/amd64,darwin/amd64" -ldflags="-X main.Version=$CIRCLE_BUILD_NUM" .

if [[ "$CIRCLE_BRANCH" = "master" ]]; then
	aws s3 cp mytool-darwin-10.6-amd64 s3://mybucket/mytool/mytool-darwin-amd64 --acl authenticated-read
	aws s3 cp mytool-linux-amd64 s3://mybucket/mytool/mytool-linux-amd64 --acl authenticated-read
	echo -n $CIRCLE_BUILD_NUM > VERSION && aws s3 cp VERSION  s3://mybucket/mytool/VERSION --acl authenticated-read
fi

Example

package main

import (
	"github.com/heetch/s3update"
)

var (
	// This gets set during the compilation. See below.
	Version = ""
)

func main() {
	err := s3update.AutoUpdate(s3update.Updater{
		CurrentVersion: Version,
		S3Bucket:       "mybucket",
		S3Region:       "eu-west-1",
		S3ReleaseKey:   "mytool/mytool-{{OS}}-{{ARCH}}",
		S3VersionKey:   "mytool/VERSION",
	})

  if err != nil {
    // ...
  }

  ...
}

Binary must be compiled with a flag specifying the new version:

go build -ldflags "-X main.Version=111" main.go

Contributions

Any contribution is welcomed!

  • Open an issue if you want to discuss bugs/features
  • Open Pull Requests if you want to contribute to the code

Copyright

Copyright © 2016 Heetch

See the LICENSE (MIT) file for more details.

Owner
Heetch
Enjoy your night out
Heetch
Comments
  • Use os.Executable in favor of osext.Executable

    Use os.Executable in favor of osext.Executable

    Since Go 1.8 os.Executable is part of the stdlib, so osext can be dropped from the dependencies.

    That change is not tested, because a test requires to create a s3 bucket and a fake binary. That said I'm confident because a recent change in osext invokes os.Executable if the library is compiled under Go 1.8.

  • Fix upgrade on linux systems

    Fix upgrade on linux systems

    Previously the still open executable file was directly written to, which caused the program to abort with a text file busy message on some systems.

    Now the following approach is taken:

    • Move the old file to a backup path
    • Try to write to the original path
    • If opening/writing the original path fails: Move backup to old location
    • If upgrade is successful: Remove backup file

    Based on https://github.com/exercism/cli/pull/290

  • Reduce memory consumption

    Reduce memory consumption

    Instead of copying the downloaded update into memory and then writing from memory to the final file, the reader for the download is used for buffered writing to the final file.


    I tested it with a "hello world" program that compiles to a 10 MB executable and then using pprof to create a memory profile. For creating the profile I imported github.com/pkg/profile into the s3updater package and called profile.Start(profile.MemProfile).Stop() right before the os.Exit(0). I profiled the code multiple times and the resulting profiles weren't always the same, but each time the original s3updater code lead to at least 8 MB memory usage, while the code after my changes always lead to less than 110 KB. For example:

    Original s3updater | After my changes -----------------------|------------------- memory profile | memory profile

    I only used Go profiling a few times before and only CPU profiling, so maybe there's something wrong about this or it doesn't show the full picture of what's happening. But I also checked my PC's memory usage and the difference can be seen in the same way.

    Now, 10 MB RAM is not a big deal. But some Go programs use packages like github.com/gobuffalo/packr (or one of its many alternatives that are compared here), so the file to update could be 100 MB or more. When running on a memory-constrained device like a Raspberry Pi Zero W (with only 512 MB RAM), this would be bad.

    Additionally, in the original implementation the garbage collector doesn't know that it can release the allocated memory, because when s3udpater downloads the update and runs it, it blocks in cmd.Run() and data still references the downloaded bytes and the GC doesn't know if it will still be accessed or not. This means the memory usage isn't just high during the update, but for all remaining time the updated program is running. Maybe this can already be mitigated by moving the part with the download and file operations into a function, so the GC sees that there's no reference to the slice of bytes any more after the function returned. But that would still lead to the temporary high memory usage during the update. With the buffered write both is covered.

    Let me know what you think :)

  • Replace exec.Command with syscall.Exec

    Replace exec.Command with syscall.Exec

    Instead of spawning child processes you should replace the parent process. You could do this by replacing the exec.Command block on line 156-177 with this line: syscall.Exec(os.Args[0], os.Args, os.Environ())

    Related to exec: https://groob.io/posts/golang-execve/

  • Save download to temporary file

    Save download to temporary file

    Instead of moving current to a backup file, save download to temporary file.

    This will prevent issue where update is cancelled by external factor, where we would not be able to move .bak to its original location.

  • DELIVERY-225 fix file not found

    DELIVERY-225 fix file not found

    Following DELIVERY-213 we didn't fixed the error

    The argument passed to syscall.Exec (os.Args[0]) suppose relative binary will be found/resolved properly

    It succeeded with ./releases/aww-darwin-amd64 and /usr/local/bin/aww but failed with path call aww

    Using absolute destination enforce the absolute resolution and hence resolves the error for further releases

  • Add autorestart mechanism after an update

    Add autorestart mechanism after an update

    With the current implementation each update requiers the user to re-run the command.

    This PR solves this by restarting the command automatically, so the user doesn't need to do anything except waiting for the process to complete 🤘

  • Prevent broken binary if download process is Interrupted

    Prevent broken binary if download process is Interrupted

    Currently, if something happen during the download process, the binary file is corrupted and need to be manually restored.

    I propose doing a very simple verification by checking the downloaded file size after downloading has finished, then proceed.

  • Add tests

    Add tests

    Currently only tests for errors.

    A test for a working update via S3 would require a mocked S3 that's being started and stopped from the test. I think that can be done, but maybe in a later PR.

Simple plugin to enable the /flip command inside of Mattermost

Plugin Starter Template This plugin serves as a starting point for writing a Mattermost plugin. Feel free to base your own plugin off this repository.

Nov 4, 2021
Demo on how an executable can respawn after an update

Auto respawn on update demo Demo on how an executable can respawn after an update How to build go build updatedemo.go How to run ./updatedemo Rebuil

Nov 2, 2021
An easy way to add useful startup banners into your Go applications
An easy way to add useful startup banners into your Go applications

Try browsing the code on Sourcegraph! Banner Add beautiful banners into your Go applications Table of Contents Motivation Usage API Command line flags

Jan 1, 2023
Embedded, self-hosted swagger-ui for go servers

swaggerui Embedded, self-hosted Swagger Ui for go servers This module provides swaggerui.Handler, which you can use to serve an embedded copy of Swagg

Dec 31, 2022
Universal code search (self-hosted)
Universal code search (self-hosted)

Sourcegraph OSS edition is a fast, open-source, fully-featured code search and navigation engine. Enterprise editions are available. Features Fast glo

Jan 9, 2023
Self hosted search engine for data leaks and password dumps
Self hosted search engine for data leaks and password dumps

Self hosted search engine for data leaks and password dumps. Upload and parse multiple files, then quickly search through all stored items with the power of Elasticsearch.

Aug 2, 2021
A framework for constructing self-spreading binaries
A framework for constructing self-spreading binaries

A framework that aids in creation of self-spreading software Requirements go get -u github.com/redcode-labs/Coldfire go get -u github.com/yelinaung/go

Jan 2, 2023
Listmonk - a standalone, self-hosted, newsletter and mailing list manager
Listmonk - a standalone, self-hosted, newsletter and mailing list manager

listmonk is a standalone, self-hosted, newsletter and mailing list manager. It is fast, feature-rich, and packed into a single binary. It uses a Postg

Jan 13, 2022
An easy to use, extensible health check library for Go applications.

Try browsing the code on Sourcegraph! Go Health Check An easy to use, extensible health check library for Go applications. Table of Contents Example M

Dec 30, 2022
A simple wrapper to daemonize Go applications.

daemonigo A simple library to daemonize Go programming language applications. Installing $ go get github.com/tyranron/daemonigo After this command da

Jul 15, 2022
Prometheus instrumentation library for Go applications

Prometheus Go client library This is the Go client library for Prometheus. It has two separate parts, one for instrumenting application code, and one

Jan 3, 2023
A simple Cron library for go that can execute closures or functions at varying intervals, from once a second to once a year on a specific date and time. Primarily for web applications and long running daemons.

Cron.go This is a simple library to handle scheduled tasks. Tasks can be run in a minimum delay of once a second--for which Cron isn't actually design

Dec 17, 2022
A BPMN engine, meant to be embedded in Go applications with minim hurdles, and a pleasant developer experience using it.

A BPMN engine, meant to be embedded in Go applications with minim hurdles, and a pleasant developer experience using it. This approach can increase transparency for non-developers.

Dec 29, 2022
A simple package to daemonize Go applications.

A simple package to daemonize Go applications.

Nov 13, 2021
Chaosblade executor for chaos experiments on Java applications
Chaosblade executor for chaos experiments on Java applications

Chaosblade-exec-jvm: Chaosblade executor for chaos experiments on Java applications Introduction The project is a chaosblade executor based on jvm-san

Dec 16, 2022
James is your butler and helps you to create, build, debug, test and run your Go projects
James is your butler and helps you to create, build, debug, test and run your Go projects

go-james James is your butler and helps you to create, build, debug, test and run your Go projects. When you often create new apps using Go, it quickl

Oct 8, 2022
:guardsman: A teeny tiny and somewhat opinionated generator for your next golang project

A Yeoman Golang Generator We are very sorry Gophers, but other names for the generator where taken, so we choose go-lang. But we have gocreate as an a

Sep 27, 2022
Auto-evaluate your Golang code.
Auto-evaluate your Golang code.

Ginker Ginker is a GUI application for auto-evaluating your Golang code. It allows you to write and run Golang code on the fly and it will help you to

Jun 24, 2021
:mailbox_closed: Your own local SMS gateway in Go
:mailbox_closed: Your own local SMS gateway in Go

gosms Your own local SMS gateway What's the use ? Can be used to send SMS, where you don't have access to internet or cannot use Web SMS gateways or w

Jan 2, 2023