Statigz serves pre-compressed embedded files with http in Go

statigz

Build Status Coverage Status GoDevDoc Time Tracker Code lines Comments

statigz serves pre-compressed embedded files with http in Go 1.16 and later.

Why?

Since version 1.16 Go provides standard way to embed static assets. This API has advantages over previous solutions:

  • assets are processed during build, so there is no need for manual generation step,
  • embedded data does not need to be kept in residential memory (as opposed to previous solutions that kept data in regular byte slices).

A common case for embedding is to serve static assets of a web application. In order to save bandwidth and improve latency, those assets are often served compressed. Compression concerns are out of embed responsibilities, yet they are quite important. Previous solutions (for example vfsgen with httpgzip) can optimize performance by storing compressed assets and serving them directly to capable user agents. This library implements such functionality for embedded file systems.

Read more in a blog post.

NOTE: Guarding new api (embed) with build tags is not a viable option, since it imposes issue in older versions of Go.

Example

package main

import (
	"embed"
	"log"
	"net/http"

	"github.com/vearutop/statigz"
	"github.com/vearutop/statigz/brotli"
)

// Declare your embedded assets.

//go:embed static/*
var st embed.FS

func main() {
	// Plug static assets handler to your server or router.
	err := http.ListenAndServe(":80", statigz.FileServer(st, brotli.AddEncoding))
	if err != nil {
		log.Fatal(err)
	}
}

Usage

Behavior is based on nginx gzip static module and github.com/lpar/gzipped.

Static assets have to be manually compressed with additional file extension, e.g. bundle.js would become bundle.js.gz (compressed with gzip) or index.html would become index.html.br (compressed with brotli).

NOTE: zopfli provides better compression than gzip while being backwards compatible with it.

Upon request server checks if there is a compressed file matching Accept-Encoding and serves it directly.

If user agent does not support available compressed data, server uses an uncompressed file if it is available ( e.g. bundle.js). If uncompressed file is not available, then server would decompress a compressed file into response.

Responses have ETag headers (64-bit FNV-1 hash of file contents) to enable caching. Responses that are not dynamically decompressed are served with http.ServeContent for ranges support.

Brotli support

Support for brotli is optional. Using brotli adds about 260 KB to binary size, that's why it is moved to a separate package.

NOTE: Although brotli has better compression than gzip and already has wide support in browsers, it has limitations for non-https servers, see this and this.

Runtime encoding

Recommended way of embedding assets is to compress assets before the build, so that binary includes *.gz or *.br files. This can be inconvenient in some cases, there is EncodeOnInit option to compress assets in runtime when creating file server. Once compressed, assets will be served directly without additional dynamic compression.

Files with extensions ".gz", ".br", ".gif", ".jpg", ".png", ".webp" are excluded from runtime encoding by default.

NOTE: Compressing assets in runtime can degrade startup performance and increase memory usage to prepare and store compressed data.

Mounting a subdirectory

It may be convenient to strip leading directory from an embedded file system, you can do that with fs.Sub and a type assertion.

package main

import (
	"embed"
	"io/fs"
	"log"
	"net/http"

	"github.com/vearutop/statigz"
	"github.com/vearutop/statigz/brotli"
)

// Declare your embedded assets.

//go:embed static/*
var st embed.FS

func main() {
	// Retrieve sub directory.
	sub, err := fs.Sub(st, "static")
	if err != nil {
		log.Fatal(err)
	}

	// Plug static assets handler to your server or router.
	err = http.ListenAndServe(":80", statigz.FileServer(sub.(fs.ReadDirFS), brotli.AddEncoding))
	if err != nil {
		log.Fatal(err)
	}
}
Owner
Viacheslav Poturaev
Born in the snows of Siberia, wandering around the world, seeking the gold.
Viacheslav Poturaev
Comments
  • index.html not compressed when served from

    index.html not compressed when served from "/" and EncodedOnInit is set

    Love this package! Very easy to use and a great help.

    I noticed that when you turn on EncodeOnInit the assets you would expect are compressed, except for the index.html when served from /.

  • Serve and redirect index pages like http.FileServer

    Serve and redirect index pages like http.FileServer

    Resolves https://github.com/vearutop/statigz/issues/9.

    This PR alters handling of redirections and serving of index pages to align with standard http.FileServer.

  • Add Found method to check if resource is available

    Add Found method to check if resource is available

    Related to #17.

    func ExampleServer_Found() {
    	fileServer := statigz.FileServer(st)
    	customHandler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    		// Serve existing static resource.
    		if fileServer.Found(r) {
    			fileServer.ServeHTTP(w, r)
    
    			return
    		}
    
    		// Do something custom for non-existing resource, for example serve index page.
    		// (This is an example, serving index instead of 404 might not be the best idea in real life 😅).
    		r.URL.Path = "/"
    		fileServer.ServeHTTP(w, r)
    	})
    
    	// Plug static assets handler to your server or router.
    	if err := http.ListenAndServe("localhost:80", customHandler); err != nil {
    		log.Fatal(err)
    	}
    }
    
  • SPA support?

    SPA support?

    I am developing a single page application and would like to use client side routing. If a static resource was not found it should always return the content of the index.html. Would this functionality be possible to add?

  • Serving index.html

    Serving index.html

    It doesn't look like statigz is serving index.html properly when it's left off the URL. (I am using the Mounting a subdirectory option from the README in case that's an issue.)

    e.g. http://localhost/ should be serving index.html, but it's not.

    http.FileServer will serve this properly.

    In fact, it says in the http.FileServer docs:

    As a special case, the returned file server redirects any request ending in "/index.html" to the same path, without the final "index.html".

    ...and then the code serving the files in http/fs.go has some checks to default to index.html in a dir.

    It would be useful for statigz to handle this the same way so it can be a drop-in replacement.

    Thanks!

  • How to work with gin?

    How to work with gin?

    Thanks for this cool library! I am using gin to build a web server, and gin has its own way to build a static server:

     r.StaticFS("/static", http.Dir("./static"))
    

    Is there any way to use statigz with gin?

  • Fix JS Content-Type in some Windows environments

    Fix JS Content-Type in some Windows environments

    We ran into this little bug while deploying statigz. Some unidentified but popular Windows software sets registry entries with incorrect .js MIME-type. This sets the official .js mime-type as an override.

    Also see https://github.com/golang/go/issues/32350

    // TypeByExtension returns the MIME type associated with the file extension ext.
    // The extension ext should begin with a leading dot, as in ".html".
    ...
    // On Windows, MIME types are extracted from the registry.
    
  • Using this library with fs.Sub

    Using this library with fs.Sub

    I am trying to use this library with fs.Sub as my assets are in a subfolder. But this library requires fs.ReadDirFS

    Is there any better approach that I do not know off or are changes to the library required ?

    Greetings

  • Implement an option to compress assets at init

    Implement an option to compress assets at init

    Having uncompressed assets can be necessary during development, having both uncompressed and compressed assets checked in the repo may lead to data sync issues for absence of single source of truth.

    A workaround could be to embed uncompressed assets and compress them during init, it would have startup latency and runtime memory usage penalty as a price for convenience of having a single file for each asset.

Composable chains of nested http.Handler instances.

chain go get github.com/codemodus/chain Package chain aids the composition of nested http.Handler instances. Nesting functions is a simple concept. I

Sep 27, 2022
Go http.Hander based middleware stack with context sharing

wrap Package wrap creates a fast and flexible middleware stack for http.Handlers. Features small; core is only 13 LOC based on http.Handler interface;

Apr 5, 2022
Minimalist net/http middleware for golang

interpose Interpose is a minimalist net/http middleware framework for golang. It uses http.Handler as its core unit of functionality, minimizing compl

Sep 27, 2022
Add interceptors to GO http.Client

mediary Add interceptors to http.Client and you will be able to Dump request and/or response to a Log Alter your requests before they are sent or resp

Nov 17, 2022
Lightweight Middleware for net/http

MuxChain MuxChain is a small package designed to complement net/http for specifying chains of handlers. With it, you can succinctly compose layers of

Dec 10, 2022
Idiomatic HTTP Middleware for Golang

Negroni Notice: This is the library formerly known as github.com/codegangsta/negroni -- Github will automatically redirect requests to this repository

Jan 2, 2023
A tiny http middleware for Golang with added handlers for common needs.

rye A simple library to support http services. Currently, rye provides a middleware handler which can be used to chain http handlers together while pr

Jan 4, 2023
A collection of useful middleware for Go HTTP services & web applications 🛃

gorilla/handlers Package handlers is a collection of handlers (aka "HTTP middleware") for use with Go's net/http package (or any framework supporting

Dec 31, 2022
Simple middleware to rate-limit HTTP requests.

Tollbooth This is a generic middleware to rate-limit HTTP requests. NOTE 1: This library is considered finished. NOTE 2: Major version changes are bac

Dec 28, 2022
A HTTP mole service
A HTTP mole service

httpmole provides a HTTP mock server that will act as a mole among your services, telling you everything http clients send to it and responding them whatever you want it to respond. Just like an actual mole.

Jul 27, 2022
Mahi is an all-in-one HTTP service for file uploading, processing, serving, and storage.
Mahi is an all-in-one HTTP service for file uploading, processing, serving, and storage.

Mahi is an all-in-one HTTP service for file uploading, processing, serving, and storage. Mahi supports chunked, resumable, and concurrent uploads. Mahi uses Libvips behind the scenes making it extremely fast and memory efficient.

Dec 29, 2022
HTTP/2 Apple Push Notification service (APNs) provider for Go with token-based connection

APNs Provider HTTP/2 Apple Push Notification service (APNs) provider for Go with token-based connection Example: key, err := apns.AuthKeyFromFile("Aut

Dec 29, 2022
OpenID Connect (OIDC) http middleware for Go

Go OpenID Connect (OIDC) HTTP Middleware Introduction This is a middleware for http to make it easy to use OpenID Connect. Currently Supported framewo

Jan 1, 2023
Hex - Expectations for HTTP handlers

Hex is a simple wrapper that extends httptest.Server with an expectation syntax, allowing you to create mock APIs using a simple and expressive DSL:

Aug 9, 2021
A Concurrent HTTP Static file server using golang .

A Concurrent HTTP static server using Golang. Serve Static files like HTML,CSS,Js,Images,Videos ,ect. using HTTP. It is Concurrent and Highly Scalable.Try now!

Dec 19, 2021
Go HTTP middleware to filter clients by IP

Go HTTP middleware to filter clients by IP

Oct 30, 2022
A dead simple, stupid, http service.

A dead simple, stupid, http service implemented in a complicated way just for the sake of following Go design patterns and scalability. Useful for learning and testing basic kubernetes networking. Made on an insomniac night.

Sep 2, 2022
A http service to verify request and bounce them according to decisions made by CrowdSec.

traefik-crowdsec-bouncer A http service to verify request and bounce them according to decisions made by CrowdSec. Description This repository aim to

Dec 21, 2022
Chi ip banner is a chi middleware that bans some ips from your Chi http server.

Chi Ip Banner Chi ip banner is a chi middleware that bans some ips from your Chi http server. It reads a .txt file in your project's root, called bani

Jan 4, 2022