Using brotli compression to embed static files in Go.

🥦 Broccoli

go get -u aletheia.icu/broccoli

GoDoc Travis Go Report Card codecov.io

Broccoli uses brotli compression to embed a virtual file system of static files inside Go executables.

A few reasons to pick broccoli over the alternatives:

  • ⚡️ The average is 13-25% smaller binary size due to use of superior compression algorithm, brotli.
  • 💾 Broccoli supports bundling of multiple source directories, only relies on go generate command-line interface and doesn't require configuration files.
  • 🔑 Optional decompression is something you may want; when it's enabled, files are decompressed only when they are read the first time.
  • 🚙 You might want to target wasm/js architecture.
  • 📰 There is -gitignore option to ignore files, already ignored by your existing .gitignore files.

Performance

Admittedly, there are already many packages providing similar functionality out there in the wild. Tim Shannon did an overall pretty good overview of them in Choosing A Library to Embed Static Assets in Go, but it should be outdated by at least two years, so although we subscribe to the analysis, we cannot guarantee whether if it's up–to–date. Most if not all of the packages mentioned in the article, rely on gzip compression and most of them, unfortunately are not compatible with wasm/js architecture, due to some quirk that has to do with their use of http package. This, among other things, was the driving force behind the creation of broccoli.

The most feature-complete library from the comparison table seems to be fileb0x.

How does broccoli compare to flexb0x?

Feature fileb0x broccoli
compression gzip brotli (-20% avg.)
optional decompression yes yes
compression levels yes yes (1-11)
different build tags for each file yes no
exclude / ignore files glob glob
unexported vars/funcs optional optional
virtual memory file system yes yes
http file system yes yes
replace text in files yes no
glob support yes yes
regex support no no
config file yes no
update files remotely yes no
.gitignore support no yes

How does it compare to others?

Broccoli seems to outperform the existing solutions.

We did benchmarks, please feel free to review them and correct us whenever our methodology could be flawed.

Usage

$ broccoli
Usage: broccoli [options]

Broccoli uses brotli compression to embed a virtual file system in Go executables.

Options:
	-src folder[,file,file2]
		The input files and directories, "public" by default.
	-o
		Name of the generated file, follows input by default.
	-var=br
		Name of the exposed variable, "br" by default.
	-include *.html,*.css
		Wildcard for the files to include, no default.
	-exclude *.wasm
		Wildcard for the files to exclude, no default.
	-opt
		Optional decompression: if enabled, files will only be decompressed
		on the first time they are read.
	-gitignore
		Enables .gitignore rules parsing in each directory, disabled by default.
	-quality [level]
		Brotli compression level (1-11), the highest by default.

Generate a broccoli.gen.go file with the variable broccoli:
	//go:generate broccoli -src assets -o broccoli -var broccoli

Generate a regular public.gen.go file, but include all *.wasm files:
	//go:generate broccoli -src public -include="*.wasm"

How broccoli is used in the user code:

//go:generate broccoli -src=public,others -o assets

func init() {
    br.Walk("public", func(path string, info os.FileInfo, err error) error {
        // walk...
        return nil
    })
}

func main() {
    http.ListenAndServe(":8080", br.Serve("public"))
}

Credits

License: MIT

We would like to thank brotli development team from Google and Andy Balholm, for his c2go pure-Go port of the library. Broccoli itself is an effort of a mentoring experiment, lead by @tucnak on the foundation of Aletheia.

Comments
  • Inflexibility WRT Go source and VFS data locations

    Inflexibility WRT Go source and VFS data locations

    Currently, broccoli must be called from a Go source directory (I call it manually via go run aletheia.icu/broccoli and don't use Go's generation feature). If this is not the case, it errors out with broccoli: cannot parse package: no buildable Go source files in ..

    Now if the data I want to have broccoli slurp resides somewhere completely else (in my case ../../../../frontend/dist; making this absolute wouldn't help either), the path ends up in the VFS, which will rarely be desirable. Because of the requirement to run broccoli from a source directory, I cannot opt to, say, run it from the dist directory in the example and have it write a Go file to a completely different directory.

    There should be an option to map the given source directories to a (common) prefix that replaces any path implied by the physical location. For example, I'd like to make the contents of the distant directory referenced above available at the root of the VFS. If not that, an option to have it strip n levels of directories would help in my case as well.

  • File size seems bigger than packr

    File size seems bigger than packr

    Hello,

    We chatted a little on Reddit. I wanted to use what you had so far and see how much smaller is the output.

    I have a project called Dozzle. I compared the output with what I currently have with packr and the binary is actually bigger.

    See below:

    Screen Shot 2020-04-16 at 10 07 48 AM

    I have pushed my branch to broccoli. Not sure what I am doing wrong. You can run yarn build to get a static folder that is the source.

  • Go mod dependency issue

    Go mod dependency issue

    In the course of trying this (with Go v1.14.2), I encountered the following error running go mod tidy after adding broccoli as a dependency (via a side-effect-only tool import, import _ "aletheia.icu/broccoli"):

    go: finding module for package aletheia.icu/broccoli
    go: downloading aletheia.icu/broccoli v1.0.0-beta
    go: found aletheia.icu/broccoli in aletheia.icu/broccoli v1.0.0-beta
    go: aletheia.icu/[email protected] requires
            aletheia.icu/broccoli/[email protected]: invalid version: unknown revision 000000000000
    
  • 404 on downloads

    404 on downloads

    Seems when trying to run a go generate utilising this library that the following error is returned:

    go: aletheia.icu/broccoli/[email protected]: unrecognized import path "aletheia.icu/broccoli/fs": reading https://aletheia.icu/broccoli/fs?go-get=1: 404 Not Found

    I can see that https://aletheia.icu is still functional but seems the path for the library isn't working as intended.

  • Issues importing the package

    Issues importing the package

    Something happened to your hosting that the go package can't be retrieved any more.

    $ go get aletheia.io/broccoli@5bc1e2f
    go get aletheia.io/broccoli@5bc1e2f: unrecognized import path "aletheia.io/broccoli": https fetch: Get "https://aletheia.io/broccoli?go-get=1": x509: certificate is valid for blog.inkanex.com, not aletheia.io
    
    $ go get aletheia.io/broccoli@5bc1e2f: unrecognized import path "aletheia.io/broccoli": reading https://aletheia.io/broccoli?go-get=1: 401 Unauthorized```
  • Concerns about dependency size

    Concerns about dependency size

    This package uses https://github.com/andybalholm/brotli to handle the actual compression. This package has 260 thousand lines of code:

    ❯ gocloc brotli
    -------------------------------------------------------------------------------
    Language                     files          blank        comment           code
    -------------------------------------------------------------------------------
    Go                              61           2451           1941         252967
    Plain Text                       1            815              0           8471
    Markdown                         1              1              0              4
    -------------------------------------------------------------------------------
    TOTAL                           63           3267           1941         261442
    -------------------------------------------------------------------------------
    

    Almost none of that are tests, so all of that code is getting compiled in. In my testing adding this package increases my binary size by about 5.5mb. In comparison, github.com/klauspost/compress/zstd is 10 thousand lines of code with tests, and only adds about 700kb to the binary.

    I'm afraid that the increase in binary size due to https://github.com/andybalholm/brotli offsets the gains made by better compression.

  • Allow direct `go generate` usage (no CLI)

    Allow direct `go generate` usage (no CLI)

    Right now I need to modify my Makefile to install broccoli in $PATH to be able to use it from go generate. It would be simpler if I could instead invoke my own gen.go which just calls the broccoli API, like vfsgen allows. From what I can see, main.go is already a very thin wrapper, all that's needed is to export Generator options and parsePackage() / generate() functions. Is that something you'd be open to? If yes, I could send in a PR.

  • Consider moving generated file comment to satisfy golint

    Consider moving generated file comment to satisfy golint

    Because the current comment is put at the top it causes golint to complain:

    domains.gen.go:1:1: package comment should be of the form "Package swot ..."
    
    // Code generated by broccoli at 2020-04-20T23:26:30+07:00.
    package swot
    
    import "aletheia.icu/broccoli/fs"
    
    var br = fs.New("...")
    
    Screen Shot 2020-04-20 at 22 57 35

    I suggest moving it to above the variable:

    package swot
    
    import "aletheia.icu/broccoli/fs"
    
    // Code generated by broccoli at 2020-04-20T23:26:30+07:00.
    var br = fs.New("...")
    
  • Output flag not working as expected

    Output flag not working as expected

    The help shows -o flag allows you to name the generated file.

    	-o
    		Name of the generated file, follows input by default.
    

    If I use broccoli -src domains -o domains I get a file called:

    domains.gen.go

    But when I use broccoli -src domains -o domains.go I still get a file called:

    domains.gen.go

a better customizable tool to embed files in go; also update embedded files remotely without restarting the server

fileb0x What is fileb0x? A better customizable tool to embed files in go. It is an alternative to go-bindata that have better features and organized c

Dec 27, 2022
Generates go code to embed resource files into your library or executable

Deprecating Notice go is now going to officially support embedding files. The go command will support //go:embed tags. Go Embed Generates go code to e

Jun 2, 2021
Embed files into a Go executable

statik statik allows you to embed a directory of static files into your Go binary to be later served from an http.FileSystem. Is this a crazy idea? No

Dec 29, 2022
A tool to be used with 'go generate' to embed external template files into Go code.

templify A tool to be used with 'go generate' to embed external template files into Go code. Scenario An often used scenario in developing go applicat

Sep 27, 2022
:file_folder: Embeds static resources into go files for single binary compilation + works with http.FileSystem + symlinks

Package statics Package statics embeds static files into your go applications. It provides helper methods and objects to retrieve embeded files and se

Sep 27, 2022
Get an embed.FS from inside an embed.FS
Get an embed.FS from inside an embed.FS

embed.FS wrapper providing additional functionality Features Get an embed.FS from an embedded subdirectory Handy Copy(sourcePath, targetPath) method t

Sep 27, 2022
The simple and easy way to embed static files into Go binaries.

NOTICE: Please consider migrating your projects to github.com/markbates/pkger. It has an idiomatic API, minimal dependencies, a stronger test suite (t

Dec 25, 2022
Embed static files in Go binaries (replacement for gobuffalo/packr)

Pkger github.com/markbates/pkger is a tool for embedding static files into Go binaries. It will, hopefully, be a replacement for github.com/gobuffalo/

Dec 29, 2022
Pure Go Brotli encoder and decoder

This package is a brotli compressor and decompressor implemented in Go. It was translated from the reference implementation (https://github.com/google

Dec 28, 2022
Zibr - Repack ZIPs into Brotli

zibr zipr is a command-line utility that repacks a compressed ZIP or PNG file in

Oct 8, 2022
a better customizable tool to embed files in go; also update embedded files remotely without restarting the server

fileb0x What is fileb0x? A better customizable tool to embed files in go. It is an alternative to go-bindata that have better features and organized c

Dec 27, 2022
Generates go code to embed resource files into your library or executable

Deprecating Notice go is now going to officially support embedding files. The go command will support //go:embed tags. Go Embed Generates go code to e

Jun 2, 2021
Embed files into a Go executable

statik statik allows you to embed a directory of static files into your Go binary to be later served from an http.FileSystem. Is this a crazy idea? No

Dec 29, 2022
A tool to be used with 'go generate' to embed external template files into Go code.

templify A tool to be used with 'go generate' to embed external template files into Go code. Scenario An often used scenario in developing go applicat

Sep 27, 2022
Embed files into a Go executable

statik statik allows you to embed a directory of static files into your Go binary to be later served from an http.FileSystem. Is this a crazy idea? No

Jan 6, 2023
Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

Dec 29, 2022
Split multiple Kubernetes files into smaller files with ease. Split multi-YAML files into individual files.

kubectl-slice: split Kubernetes YAMLs into files kubectl-slice is a neat tool that allows you to split a single multi-YAML Kubernetes manifest into mu

Jan 3, 2023
Simple image compression using SVD

SVD image compression An implementation image compression using SVD decomposition on Go Built With Go 1.17 Gonum Compression examples Header Image Ori

Mar 30, 2022
Go package to embed the Mozilla Included CA Certificate List

rootcerts Package rootcerts provides an embedded copy of the Mozilla Included CA Certificate List, more specifically the PEM of Root Certificates in M

Oct 21, 2022
Recreate embedded filesystems from embed.FS type in current working directory.

rebed Recreate embedded filesystems from embed.FS type in current working directory. Expose the files you've embedded in your binary so users can see

Sep 27, 2022