Build Go Toolchains /w native libs for cross-compilation

gonative

Cross compiled Go binaries are not suitable for production applications because code in the standard library relies on Cgo for DNS resolution with the native resolver, access to system certificate roots, and parts of os/user.

gonative is a simple tool which creates a build of Go that can cross compile to all platforms while still using the Cgo-enabled versions of the stdlib packages. It does this by downloading the binary distributions for each platform and copying their libraries into the proper places. It sets the correct mod time so they don't get rebuilt. It also copies some auto-generated runtime files into the build as well. gonative does not modify any Go that you have installed and builds a new installaion of Go in a separate directory (the current directory by default).

Once you have a toolchain for cross-compilation, you can use tools like gox to cross-compile native builds easily.

gonative will not help you if your own packages rely on Cgo

Installation

git clone https://github.com/inconshreveable/gonative
cd gonative
make

Alternatively, you can install gonative via go get but the dependencies are not locked down.

go get github.com/inconshreveable/gonative

Running

The 'build' command will build a toolchain in a directory called 'go' in your working directory.

gonative build

To build a particular version of Go (default is 1.4):

gonative build -version=1.3.3

For options and help:

gonative build -h

How it works

gonative downloads the go source code and compiles it for your host platform. It then bootstraps the toolchain for all target platforms (but does not compile the standard library). Then, it fetches the official binary distributions for all target platforms and copies each pkg/OS_ARCH directory into the toolchain so that you will link with natively-compiled versions of the standard library. It walks all of the copied standard library and sets their modtimes so that they won't get rebuilt. It also copies some necessary auto-generated runtime source files for each platform (z*_) into the source directory to make it all work.

Example with gox:

Here's an example of how to cross-compile a project:

$ go get github.com/mitchellh/gox
$ go get github.com/inconshreveable/gonative
$ cd /your/project
$ gonative build
$ PATH=$PWD/go/bin/:$PATH gox

This isn't the most optimal way of doing things though. You only ever need one gonative-built Go toolchain. And with the proper GOPATH set up, you don't need to be in your project's working directory. I use it mostly like this:

One time only setup:

$ go get github.com/mitchellh/gox
$ go get github.com/inconshreveable/gonative
$ mkdir -p /usr/local/gonative
$ cd /usr/local/gonative
$ gonative build

Building a project:

$ PATH=/usr/local/gonative/go/bin/:$PATH gox github.com/your-name/application-name

Open Issues

  • gonative is untested on Windows

Caveats

  • no linux/arm support because there are no official builds of linux/arm
  • linux_386 binaries that use native libs depend on 32-bit libc/libpthread/elf loader. some 64-bit linux distributions might not have those installed by default
Comments
  • gonative fails to build

    gonative fails to build

    $ godep go install
    # github.com/inconshreveable/axiom
    ../axiom/axiom.go:108: too few values in struct initializer
    ../axiom/axiom.go:109: too few values in struct initializer
    ../axiom/axiom.go:110: too few values in struct initializer
    ../axiom/axiom.go:231: too few values in struct initializer
    ../axiom/axiom.go:330: too few values in struct initializer
    godep: go exit status 2
    $ go version
    go version go1.5.1 darwin/amd64
    $ godep version
    godep v3 (darwin/amd64/go1.5)
    
  • Native unpack

    Native unpack

    This patch uses the stdlib libraries to unpack archives instead of execing tar/unzip, which allows gonative to be used on systems that don't have those tools installed. Also, Windows support probably would only take a few minor changes (copy recursive, maybe some path tweaks) for someone to implement.

    Closes #9

  • Running full make.bash for bootstrapping platforms

    Running full make.bash for bootstrapping platforms

    For some reason, using go tool dist didn't work for me on an OS X host and Go 1.3.3. When cross-compiling using gox, it complained that the target platforms still needed to be bootstrapped.

    So, I changed the code to instead run ./make.bash --no-clean as described in your blog and that took care of the problem.

  • gonative build returns

    gonative build returns "Bad response for download: 404"

    Don't manage to run the following command, it fails with 404, so I guess a wrong URL is being used somewhere...

    Cheers!

    gonative build -src=/home/ubuntu/cache/go '-platforms=windows_amd64 darwin_amd64 linux_amd64' -version=go1.4
    INFO[01-23|10:05:28] building go                              version=go1.4 src=/home/ubuntu/cache/go target=/home/ubuntu/cache/gonative/go platforms="[{OS:windows Arch:amd64} {OS:darwin Arch:amd64} {OS:linux Arch:amd64}]"
    INFO[01-23|10:05:28] start download                           plat=windows_amd64 url=https://storage.googleapis.com/golang/gogo1.4.windows-amd64.zip
    INFO[01-23|10:05:28] start download                           plat=darwin_amd64 url=https://storage.googleapis.com/golang/gogo1.4.darwin-amd64-osx10.8.tar.gz
    INFO[01-23|10:05:28] start download                           plat=linux_amd64 url=https://storage.googleapis.com/golang/gogo1.4.linux-amd64.tar.gz
    INFO[01-23|10:05:29] cp -rp /home/ubuntu/cache/go /home/ubuntu/cache/gonative/go err=nil
    # Building C bootstrap tool.
    cmd/dist
    
    # Building compilers and Go bootstrap tool for host, linux/amd64.
    lib9
    libbio
    liblink
    cmd/cc
    cmd/gc
    cmd/6l
    cmd/6a
    cmd/6c
    cmd/6g
    runtime
    errors
    sync/atomic
    sync
    io
    unicode
    unicode/utf8
    unicode/utf16
    bytes
    math
    strings
    strconv
    bufio
    sort
    container/heap
    encoding/base64
    syscall
    time
    os
    reflect
    fmt
    encoding
    encoding/json
    flag
    path/filepath
    path
    io/ioutil
    log
    regexp/syntax
    regexp
    go/token
    go/scanner
    go/ast
    go/parser
    os/exec
    os/signal
    net/url
    text/template/parse
    text/template
    go/doc
    go/build
    cmd/go
    
    # Building packages and commands for linux/amd64.
    runtime
    errors
    sync/atomic
    unicode
    sync
    io
    unicode/utf8
    math
    bytes
    bufio
    syscall
    strconv
    time
    reflect
    os
    strings
    fmt
    encoding/binary
    sort
    regexp/syntax
    cmd/internal/goobj
    cmd/internal/rsc.io/arm/armasm
    cmd/internal/rsc.io/x86/x86asm
    debug/dwarf
    debug/elf
    debug/gosym
    debug/macho
    debug/pe
    debug/plan9obj
    regexp
    text/tabwriter
    flag
    log
    hash
    crypto
    cmd/internal/objfile
    crypto/md5
    go/token
    path/filepath
    go/scanner
    cmd/addr2line
    go/ast
    io/ioutil
    go/parser
    go/printer
    os/exec
    path
    cmd/cgo
    go/format
    cmd/fix
    container/heap
    encoding
    encoding/base64
    unicode/utf16
    encoding/json
    encoding/xml
    net/url
    text/template/parse
    compress/flate
    text/template
    hash/crc32
    compress/gzip
    container/list
    crypto/subtle
    crypto/cipher
    crypto/aes
    go/doc
    crypto/des
    math/rand
    go/build
    math/big
    crypto/hmac
    internal/syscall
    crypto/rc4
    crypto/sha1
    crypto/sha256
    crypto/elliptic
    encoding/asn1
    crypto/rand
    crypto/ecdsa
    crypto/rsa
    crypto/dsa
    crypto/sha512
    crypto/x509/pkix
    encoding/hex
    encoding/pem
    runtime/cgo
    mime
    net/http/internal
    os/signal
    runtime/pprof
    cmd/gofmt
    net
    cmd/nm
    cmd/objdump
    cmd/pack
    crypto/x509
    net/textproto
    mime/multipart
    crypto/tls
    cmd/pprof/internal/profile
    cmd/pprof/internal/plugin
    html
    html/template
    net/http
    cmd/pprof/internal/report
    cmd/pprof/internal/svg
    cmd/pprof/internal/tempfile
    cmd/pprof/internal/commands
    cmd/go
    cmd/pprof/internal/driver
    cmd/pprof/internal/fetch
    cmd/pprof/internal/symbolizer
    cmd/pprof/internal/symbolz
    cmd/pprof
    cmd/yacc
    archive/tar
    archive/zip
    compress/bzip2
    compress/lzw
    hash/adler32
    container/ring
    database/sql/driver
    compress/zlib
    encoding/ascii85
    database/sql
    encoding/base32
    encoding/csv
    encoding/gob
    expvar
    hash/crc64
    hash/fnv
    image/color
    image
    image/color/palette
    image/draw
    image/jpeg
    image/gif
    image/png
    index/suffixarray
    log/syslog
    math/cmplx
    net/http/cgi
    net/http/cookiejar
    net/http/httptest
    net/http/fcgi
    net/http/httputil
    net/http/pprof
    net/mail
    net/rpc
    net/smtp
    os/user
    net/rpc/jsonrpc
    runtime/debug
    runtime/race
    testing
    testing/iotest
    testing/quick
    text/scanner
    
    
    ---
    Installed Go for linux/amd64 in /home/ubuntu/cache/gonative/go
    Installed commands in /home/ubuntu/cache/gonative/go/bin
    INFO[01-23|10:06:18] boostraping go compilers 
    lib9
    libbio
    liblink
    cmd/cc
    cmd/gc
    cmd/6l
    cmd/6a
    cmd/6c
    cmd/6g
    runtime
    errors
    sync/atomic
    sync
    io
    unicode
    unicode/utf8
    unicode/utf16
    bytes
    math
    strings
    strconv
    bufio
    sort
    container/heap
    encoding/base64
    syscall
    time
    os
    reflect
    fmt
    encoding
    encoding/json
    flag
    path/filepath
    path
    io/ioutil
    log
    regexp/syntax
    regexp
    go/token
    go/scanner
    go/ast
    go/parser
    os/exec
    os/signal
    net/url
    text/template/parse
    text/template
    go/doc
    go/build
    cmd/go
    runtime (windows/amd64)
    lib9
    libbio
    liblink
    cmd/cc
    cmd/gc
    cmd/6l
    cmd/6a
    cmd/6c
    cmd/6g
    runtime
    errors
    sync/atomic
    sync
    io
    unicode
    unicode/utf8
    unicode/utf16
    bytes
    math
    strings
    strconv
    bufio
    sort
    container/heap
    encoding/base64
    syscall
    time
    os
    reflect
    fmt
    encoding
    encoding/json
    flag
    path/filepath
    path
    io/ioutil
    log
    regexp/syntax
    regexp
    go/token
    go/scanner
    go/ast
    go/parser
    os/exec
    os/signal
    net/url
    text/template/parse
    text/template
    go/doc
    go/build
    cmd/go
    runtime (darwin/amd64)
    lib9
    libbio
    liblink
    cmd/cc
    cmd/gc
    cmd/6l
    cmd/6a
    cmd/6c
    cmd/6g
    runtime
    errors
    sync/atomic
    sync
    io
    unicode
    unicode/utf8
    unicode/utf16
    bytes
    math
    strings
    strconv
    bufio
    sort
    container/heap
    encoding/base64
    syscall
    time
    os
    reflect
    fmt
    encoding
    encoding/json
    flag
    path/filepath
    path
    io/ioutil
    log
    regexp/syntax
    regexp
    go/token
    go/scanner
    go/ast
    go/parser
    os/exec
    os/signal
    net/url
    text/template/parse
    text/template
    go/doc
    go/build
    cmd/go
    CRIT[01-23|10:06:50] command failed                           err="Bad response for download: 404"
    
  • Fails to build on OS X 10.8

    Fails to build on OS X 10.8

    This may be more general, but I cannot create a successful build on 10.8 for 1.2.2 or 1.3beta1, using the standard OS X gcc (which I believe is actually clang), or gcc4.6 (from homebrew).

  • Add support for 1.5.1

    Add support for 1.5.1

    Adds support for building and bootstrapping 1.5.1.

    1.5.1 does require you to set GOROOT_BOOTSTRAP correctly in order to bootstrap the compiler from an existing Go >= 1.4 compiler. It also requires setting the GOROOT variable when bootstrapping a distribution to the target Go root path, it doesn't default to this anymore sadly, at least this we were able to set automatically, unlike the bootstrap root when is up to the user.

  • make.bash build does not install race enabled stdlib packages

    make.bash build does not install race enabled stdlib packages

    Thanks for gonative. I believe that the call to "make.bash" should be converted to "all.bash" to also install the race enabled stdlib packages.

    https://code.google.com/p/go/issues/detail?id=6479

  • Add support for Go 1.5.2

    Add support for Go 1.5.2

    @inconshreveable Added links for Go 1.5.2. Haven't tested anything really, but I presume that there was no change really between 1.5.1 and 1.5.2 in the build process...

  • Does this works for compiling go using sqlite3 package

    Does this works for compiling go using sqlite3 package

    Does this works for the go projects using sqlite3? I tried to compile and got an error

    15 errors occurred:
    --> darwin/386 error: exit status 1
    Stderr: sqlite3.go:13:2: C source files not allowed when not using cgo: sqlite3.c
    
    --> darwin/amd64 error: exit status 1
    Stderr: sqlite3.go:13:2: C source files not allowed when not using cgo: sqlite3.c
    
    --> linux/386 error: exit status 1
    Stderr: sqlite3.go:13:2: C source files not allowed when not using cgo: sqlite3.c
    
    --> linux/arm error: exit status 1
    Stderr: sqlite3.go:13:2: C source files not allowed when not using cgo: sqlite3.c
    
    --> freebsd/amd64 error: exit status 1
    Stderr: sqlite3.go:13:2: C source files not allowed when not using cgo: sqlite3.c
    
    --> freebsd/386 error: exit status 1
    Stderr: sqlite3.go:13:2: C source files not allowed when not using cgo: sqlite3.c
    
    --> openbsd/386 error: exit status 1
    Stderr: sqlite3.go:13:2: C source files not allowed when not using cgo: sqlite3.c
    
    --> windows/386 error: exit status 1
    Stderr: sqlite3.go:13:2: C source files not allowed when not using cgo: sqlite3.c
    
  • Go 1.4.2

    Go 1.4.2

    The 1.4.2 binaries for FreeBSD don’t appear to exist, so I removed it from the defaults, this should be changed when they appear or a new release comes out.

  • Fails to build, 1.3 on linux-amd64

    Fails to build, 1.3 on linux-amd64

    jb@build:/usr/local/gonative$ ~/bin/gonative -version=1.3
    Building go:
        Version: 1.3
        Source: (from internet)
        Target: /usr/local/gonative
        Platforms: [{linux 386} {linux amd64} {linux arm} {darwin 386} {darwin amd64} {windows 386} {windows amd64} {freebsd 386} {freebsd amd64}]
    Fetching Go sources from https://storage.googleapis.com/golang/go1.3.src.tar.gz
    ...
    cp -rp freebsd_386-788246966/go/pkg/freebsd_386 /usr/local/gonative/go/pkg
    cp -rp darwin_amd64-349208609/go/pkg/darwin_amd64 /usr/local/gonative/go/pkg
    Failed to build Go: Bad response for download: 404
    jb@build:/usr/local/gonative$
    

    https://gist.github.com/calmh/b835212474a3b285cdae

  • gonative can't be `go get`-ed

    gonative can't be `go get`-ed

    # github.com/inconshreveable/gonative
    ../../inconshreveable/gonative/gonative.go:67:70: cannot use nil as type bool in field value
    ../../inconshreveable/gonative/gonative.go:67:70: too few values in struct initializer
    ../../inconshreveable/gonative/gonative.go:68:99: cannot use nil as type bool in field value
    ../../inconshreveable/gonative/gonative.go:68:99: too few values in struct initializer
    ../../inconshreveable/gonative/gonative.go:69:81: cannot use nil as type bool in field value
    ../../inconshreveable/gonative/gonative.go:69:81: too few values in struct initializer
    ../../inconshreveable/gonative/gonative.go:70:176: cannot use nil as type bool in field value
    ../../inconshreveable/gonative/gonative.go:70:176: too few values in struct initializer```
  • question about deprecation of this tool and alternative

    question about deprecation of this tool and alternative

    @inconshreveable

    Hello,

    You post a comment on some other issue saying gonative should be considered as deprecated

    gonative is very, very old and almost certainly no longer necessary or the right way to do whatever it is that you're trying to do. i no longer use it and you should consider it a deprecated tool.

    For my information, what are you using now for having a cross compiling toolchain ? Or what is the right way to do ?

  • go get failed with too many errors

    go get failed with too many errors

    #$ go get github.com/inconshreveable/gonative
    # github.com/inconshreveable/gonative
    ../go/src/github.com/inconshreveable/gonative/gonative.go:67: cannot use "" (type string) as type bool in field value
    ../go/src/github.com/inconshreveable/gonative/gonative.go:67: cannot use nil as type string in field value
    ../go/src/github.com/inconshreveable/gonative/gonative.go:67: too few values in struct initializer
    ../go/src/github.com/inconshreveable/gonative/gonative.go:68: cannot use "" (type string) as type bool in field value
    ../go/src/github.com/inconshreveable/gonative/gonative.go:68: cannot use nil as type string in field value
    ../go/src/github.com/inconshreveable/gonative/gonative.go:68: too few values in struct initializer
    ../go/src/github.com/inconshreveable/gonative/gonative.go:69: cannot use "" (type string) as type bool in field value
    ../go/src/github.com/inconshreveable/gonative/gonative.go:69: cannot use nil as type string in field value
    ../go/src/github.com/inconshreveable/gonative/gonative.go:69: too few values in struct initializer
    ../go/src/github.com/inconshreveable/gonative/gonative.go:70: cannot use "" (type string) as type bool in field value
    ../go/src/github.com/inconshreveable/gonative/gonative.go:70: too many errors
    
  • Failing to cross-compile when using gonative, gox and Go 1.5.1

    Failing to cross-compile when using gonative, gox and Go 1.5.1

    I am using gonative and gox to cross-compile an executable using os/user. It has always been working, but it broke down when CircleCI upgraded their Golang to 1.5.1 I think. Now I am getting user: Current not implemented on darwin/amd64 from the cross-compiled executables. Any ideas what might be wrong now before I try to dive deep into Go 1.5 cross-compilation changes myself?

    CC @inconshreveable

  • Easier Bootstrap for Go 1.5+

    Easier Bootstrap for Go 1.5+

    Building Go 1.5+ from source requires that an existing 1.4 Go toolchain be on your machine and specified via the GOROOT_BOOTSTRAP environment variable. gonative will fail if that's not supplied right now if you're building Go1.5+. Instead, if the env var is unspecified, gonative should download a Go 1.4 installation native to your platform and use that for the bootstrap process.

a build tool for Go, with a focus on cross-compiling, packaging and deployment

goxc NOTE: goxc has long been in maintenance mode. Ever since Go1.5 supported simple cross-compilation, this tool lost much of its value. There are st

Dec 9, 2022
Cross-platform beanstalkd queue server admin console.
Cross-platform beanstalkd queue server admin console.

Overview aurora is a web-based Beanstalkd queue server console written in Go and works on macOS, Linux, and Windows machines. The main idea behind usi

Dec 30, 2022
Fast cross-platform HTTP benchmarking tool written in Go

bombardier bombardier is a HTTP(S) benchmarking tool. It is written in Go programming language and uses excellent fasthttp instead of Go's default htt

Jan 2, 2023
:rocket: Modern cross-platform HTTP load-testing tool written in Go
:rocket: Modern cross-platform HTTP load-testing tool written in Go

English | 中文 Cassowary is a modern HTTP/S, intuitive & cross-platform load testing tool built in Go for developers, testers and sysadmins. Cassowary d

Dec 29, 2022
A dead simple, no frills Go cross compile tool

Gox - Simple Go Cross Compilation Gox is a simple, no-frills tool for Go cross compilation that behaves a lot like standard go build. Gox will paralle

Jan 1, 2023
A cross platform CLI for Flyte. Written in Golang. Offers an intuitive interface to Flyte https://flytectl.readthedocs.io/en/latest/
A cross platform CLI for Flyte. Written in Golang. Offers an intuitive interface to Flyte https://flytectl.readthedocs.io/en/latest/

FlyteCTL Flyte's official command-line interface Documentation · Contribution Guide FlyteCTL was designed as a portable and lightweight command-line i

Nov 7, 2022
A cross-language distributed transaction manager
A cross-language distributed transaction manager

English Docs 跨语言分布式事务管理器 DTM是一款golang开发的分布式事务管理器,解决了跨数据库、跨服务、跨语言栈更新数据的一致性问题。 他优雅

Dec 31, 2022
Quickly cross-compile your C code
Quickly cross-compile your C code

WORK IN PROGRESS: Viceroy is a very early work in progress and is subject to breaking changes. It's also subject to not really working all that well y

Aug 19, 2022
Based on the electron Cross-platform Mini browser

Based on the electron Cross-platform Mini browser

May 1, 2022
Defalyzer: A cross-platform software that focuses on defacement analyzing and mirror tracking
Defalyzer: A cross-platform software that focuses on defacement analyzing and mirror tracking

Project Defalyzer is a cross-platform software that focuses on defacement analyz

Jul 26, 2022
The Cloud Native Application Proxy
The Cloud Native Application Proxy

Traefik (pronounced traffic) is a modern HTTP reverse proxy and load balancer that makes deploying microservices easy. Traefik integrates with your ex

Jan 9, 2023
Kubernetes Operator for a Cloud-Native OpenVPN Deployment.

Meerkat is a Kubernetes Operator that facilitates the deployment of OpenVPN in a Kubernetes cluster. By leveraging Hashicorp Vault, Meerkat securely manages the underlying PKI.

Jan 4, 2023
Open Service Mesh (OSM) is a lightweight, extensible, cloud native service mesh that allows users to uniformly manage, secure, and get out-of-the-box observability features for highly dynamic microservice environments.
Open Service Mesh (OSM) is a lightweight, extensible, cloud native service mesh that allows users to uniformly manage, secure, and get out-of-the-box observability features for highly dynamic microservice environments.

Open Service Mesh (OSM) Open Service Mesh (OSM) is a lightweight, extensible, Cloud Native service mesh that allows users to uniformly manage, secure,

Jan 2, 2023
Litmus helps Kubernetes SREs and developers practice chaos engineering in a Kubernetes native way.
Litmus helps Kubernetes SREs and developers practice chaos engineering in a Kubernetes native way.

Litmus Cloud-Native Chaos Engineering Read this in other languages. ???? ???? ???? ???? Overview Litmus is a toolset to do cloud-native chaos engineer

Jan 1, 2023
Kubernetes Native Policy Management
Kubernetes Native Policy Management

Kyverno Kubernetes Native Policy Management Kyverno is a policy engine designed for Kubernetes. It can validate, mutate, and generate configurations u

Jan 2, 2023
Kubernetes Native Serverless Framework
Kubernetes Native Serverless Framework

kubeless is a Kubernetes-native serverless framework that lets you deploy small bits of code without having to worry about the underlying infrastructu

Dec 25, 2022
Zadig is a cloud native, distributed, developer-oriented continuous delivery product.

Zadig Developer-oriented Continuous Delivery Product English | 简体中文 Table of Contents Zadig Table of Contents What is Zadig Quick start How to use? Ho

Jan 8, 2023
Zadig is a cloud native, distributed, developer-oriented continuous delivery product.

Zadig Developer-oriented Continuous Delivery Product ⁣ English | 简体中文 Table of Contents Zadig Table of Contents What is Zadig Quick start How to use?

May 12, 2021
Interactive Cloud-Native Environment Client
Interactive Cloud-Native Environment Client

Fenix-CLI:Interactive Cloud-Native Environment Client English | 简体中文 Fenix-CLI is an interactive cloud-native operating environment client. The goal i

Dec 15, 2022