A simple file embedder for Go

esc

GoDoc

esc embeds files into go programs and provides http.FileSystem interfaces to them.

It adds all named files or files recursively under named directories at the path specified. The output file provides an http.FileSystem interface with zero dependencies on packages outside the standard library.

Installation

go get -u github.com/mjibson/esc

Usage

esc [flag] [name ...]

The flags are:

-o=""
	output filename, defaults to stdout
-pkg="main"
	package name of output file, defaults to main
-prefix=""
	strip given prefix from filenames
-ignore=""
	regular expression for files to ignore
-include=""
	regular expression for files to include
-modtime=""
	Unix timestamp to override as modification time for all files
-private
	unexport functions by prefixing them with esc, e.g. FS -> escFS
-no-compress
	do not compress files

Accessing Embedded Files

After producing an output file, the assets may be accessed with the FS() function, which takes a flag to use local assets instead (for local development).

  • (_esc)?FS(Must)?(Byte|String) returns an asset as a (byte slice|string).
  • (_esc)?FSMust(Byte|String) panics if the asset is not found.

Go Generate

esc can be invoked by go generate:

//go:generate esc -o static.go -pkg server static

Example

Embedded assets can be served with HTTP using the http.FileServer. Assuming you have a directory structure similar to the following:

.
├── main.go
└── static
    ├── css
    │   └── style.css
    └── index.html

Where main.go contains:

package main

import (
	"log"
	"net/http"
)

func main() {
	// FS() is created by esc and returns a http.Filesystem.
	http.Handle("/static/", http.FileServer(FS(false)))
	log.Fatal(http.ListenAndServe(":8080", nil))
}

  1. Generate the embedded data: esc -o static.go static
  2. Start the server: go run main.go static.go
  3. Access http://localhost:8080/static/index.html to view the files.

You can see worked example in example dir just run it as go run example/main.go example/static.go

Comments
  • Add support for private esc function names

    Add support for private esc function names

    Add a -private command option to generate package private function names. Standard esc exported functions will be prefixed by esc.

    Default behavior remains package exported.

  • Readdir Implementation - and covering with tests

    Readdir Implementation - and covering with tests

    • [X] Added Readdir
    • [X] Moved all string manipulations to html/template
    • [X] Cover embed.go with tests (about 80%)
    • [X] Cover generated static.go with test (about 80%)
      1. htttest for checking FileServer
      2. checking both mode - useLocal=false and useLocal=true for the same testcases
      3. added benchmark
    • [X] Added testdata dir with some static files (some html5 template)
    • [X] Added example folder with http.FileServer and generated from testdata
    • [x] Added golang.org/x/tools/imports for generated file (like goftm for generated file)
    • [X] Added to travis go generate and go test after it, as smoke tests.
  • Feature idea: Directory contents

    Feature idea: Directory contents

    Let me preface my feature request by first stating that I really have enjoyed using esc. It is simple, uses the Go toolchain (generate), and makes me happy.

    I have run across a use case where a possible feature enhancement could make this tool a little easier to use for specific use cases. Below is a sample of some code that demonstrates a scenario where I have implemented a renderer for use in the Echo framework. Simply put it provides a Render method that tells the framework how to render requests.

    In the LoadTemplates method is where the imaginary code comes in. In here I imagine the ability to do something similar to ioutil.ReadDir() and get a slice of FileInfo structs. This way I can then iterate over the files and compile them as templates.

    package ui
    
    import (
    	"fmt"
    	"html/template"
    	"io"
    	"path/filepath"
    	"strings"
    
    	"myproject/app/compiledassets"
    	"github.com/labstack/echo"
    )
    
    var templates map[string]*template.Template
    
    /*
    TemplateRenderer describes a handlers for rendering layouts/pages
    */
    type TemplateRenderer struct {
    	templates *template.Template
    }
    
    /*
    NewTemplateRenderer creates a new struct
    */
    func NewTemplateRenderer(debugMode bool) *TemplateRenderer {
    	result := &TemplateRenderer{}
    	result.LoadTemplates(debugMode)
    
    	return result
    }
    
    /*
    Implements render interface expected by Echo 
    */
    func (t *TemplateRenderer) Render(w io.Writer, name string, data interface{}, ctx echo.Context) error {
    	var tmpl *template.Template
    	var ok bool
    
    	if tmpl, ok = templates[name]; !ok {
    		return fmt.Errorf("Cannot find template %s", name)
    	}
    
    	return tmpl.ExecuteTemplate(w, "layout", data)
    }
    
    /*
    Traverses the asset pages directory and compiles templates into a package level variable
    */
    func (t *TemplateRenderer) LoadTemplates(debugMode bool) {
    	templates = make(map[string]*template.Template)
    
    	/*
    	 * This code isn't real. It is what I imagine esc could look like
    	 * to get a directory of assets. Perhaps returns a slice of 
    	 * os.FileInfo structs
    	 */
    	files := compiledassets.DirContents("/www")
    
    	for _, file := range files {
    		if !file.IsDir() {
    			basename := file.Name()
    			trimmedName := strings.TrimSuffix(basename, filepath.Ext(basename))
    
    			templates["mainLayout:"+trimmedName], _ = template.Must(
    				template.New("layout").Parse(www.FSMustString(debugMode, "/www/myproject/layouts/mainLayout.gohtml")),
    			).Parse(www.FSMustString(debugMode, "/www/myproject/pages/"+basename))
    		}
    	}
    }
    

    What are your thoughts on this? Thanks for reading!

  • Strip leading / from local path of directories

    Strip leading / from local path of directories

    Was reading system's filesystem root for directories instead of go proc's cwd. Bug was evidenced by "get /" not being translated get /index.html and also If you look at the generated static.go (or whatever) file, all directories have 'local' starting with /

    Thanks for Esc!

  • can't serve index.html

    can't serve index.html

    $ esc -o static.go index.html style.css
    
    http.Handle("/", http.FileServer(FS(Debug)))
    

    I can access /style.css but not /index.html.

    [amitu@AmitUs-MacBook-Air martd (master ✗)]$ curl -I localhost:54321/index.html
    HTTP/1.1 301 Moved Permanently
    Location: ./
    Date: Mon, 24 Nov 2014 08:29:07 GMT
    Content-Type: text/plain; charset=utf-8
    
    [amitu@AmitUs-MacBook-Air martd (master ✗)]$ curl -I localhost:54321/
    HTTP/1.1 404 Not Found
    Content-Type: text/plain; charset=utf-8
    Date: Mon, 24 Nov 2014 08:29:15 GMT
    Content-Length: 19
    
    [amitu@AmitUs-MacBook-Air martd (master ✗)]$ curl -I localhost:54321/style.css
    HTTP/1.1 200 OK
    Accept-Ranges: bytes
    Content-Length: 1378
    Content-Type: text/css; charset=utf-8
    Date: Mon, 24 Nov 2014 08:31:26 GMT
    
    [amitu@AmitUs-MacBook-Air martd (master ✗)]$
    
  • esc package imports embed locally

    esc package imports embed locally

    I see the reasoning behind this commit https://github.com/mjibson/esc/commit/ce63115657a87bf5a798fb557e2db659742e509a#diff-7ddfb3e035b42cd70649cc33393fe32cR6

    and all the ways it's used so far https://github.com/search?q=%22github.com%2Fmjibson%2Fesc%2Fembed%22&type=Code&utf8=%E2%9C%93

    But I figured for the base esc package, it made sense to import locally rather than with the github URL. This allows me to go run the esc/main.go package in some //go:generate commands where I've vendored esc ... Otherwise it complains about not finding esc in any of my path/bin ... which kinda breaks the spirit of vendoring in my use case.

    Didn't seem like this change would preclude others from using ...esc/embed as a library on its own, but I could be wrong.

  • How does this differ from other similar projects?

    How does this differ from other similar projects?

    Hi there,

    I just discovered this project. I have a similar one called vfsgen, that also generates a .go file that implements http.FileSystem and only uses standard library.

    In its README, I have a list of alternatives because I think it's a good idea, and I'd like to add your project there. Can you help me understand in what ways this project is unique, what does it focus on/optimize for, and what would be a good description?

    Thanks!

  • Add FS(Must)?(Byte|String) helper functions.

    Add FS(Must)?(Byte|String) helper functions.

    Hi Matt,

    I like esc better than all the alternatives, and here are some helper functions to get assets directly.

    Use case: storing more complex templates in external files that are going to be embedded, e.g.

    pageTemplateStr := FSMustString(false, "/page.html")
    pageTemplate = template.Must(template.New("page").Parse(pageTemplateStr))
    

    If you don't like the names, feel free to change to whatever you think is best.

  • local file not being served

    local file not being served

    $ esc -o static.go static/
    

    Generates:

    var data = map[string]*file{
        "/static/mart.js": {compressed: "....", size: 519, local: "static/static/mart.js"},
    }
    

    Which is wrong as I do not have static/static/mart.js, and hence it fails.

  • [question] will this repository suddenly disappear?

    [question] will this repository suddenly disappear?

    Hi,

    Thank you so much for this project.

    Many of us felt much pain when go-bindata unexpectedly went down and hundreds of projects stopped building. This is to verify that there are no plans for esc to repeat the same trick.

  • Making core logic availible as library

    Making core logic availible as library

    This may help solve a few different pain points with distributing and using esc. Currently we are invoking esc with //go:generate esc -modtime 0 -o static.go ...

    This works, but depends on esc being present on the user's system, and being the same version. For the ci server we have to fetch it each time the build runs (in travis, for example).

    This pr should not change the behavior of the app at all. What it does it pull all relevant logic into a separate package that can be imported by other code, and vendored appropriately. The main package now only exists to handle flags and invoke the embed package.

    I can make a generate.go file that includes it and runs it, and do //go:generate go run build/generate.go instead of invoking esc directly. Now all users, and the ci server will use the vendored copy of esc instead of depending on whatever has been installed independently.

    Thoughts?

  • security vulnerability in embedded example jquery.min.js

    security vulnerability in embedded example jquery.min.js

    Hey,

    This is the lowest of priorities I know... I have recently started using a library scanner that detects vulnerabilities in libraries that I use. It seems to have caught on to the usage of the embedded jquery in example/static.go

    Would it be possible to update the example to something else?

  • Data race on _escFile.name

    Data race on _escFile.name

    I hit this data race in my CI:

    ==================
    WARNING: DATA RACE
    Read at 0x00c0000d2a68 by goroutine 7:
      github.com/hortbot/hortbot/internal/db/migrations/esc.(*_escFile).Name()
          /home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/esc/esc.go:130 +0x3e
      github.com/golang-migrate/migrate/v4/source/httpfs.(*PartialDriver).Init()
          /home/travis/gopath/pkg/mod/github.com/golang-migrate/migrate/[email protected]/source/httpfs/partial_driver.go:48 +0x34c
      github.com/golang-migrate/migrate/v4/source/httpfs.New()
          /home/travis/gopath/pkg/mod/github.com/golang-migrate/migrate/[email protected]/source/httpfs/driver.go:21 +0xaf
      github.com/hortbot/hortbot/internal/db/migrations.newMigrate()
          /home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/migrations.go:51 +0x77
      github.com/hortbot/hortbot/internal/db/migrations.Up()
          /home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/migrations.go:15 +0x8d
      github.com/hortbot/hortbot/internal/db/migrations_test.withDatabase()
          /home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/migrations_test.go:67 +0x11e
      github.com/hortbot/hortbot/internal/db/migrations_test.TestUp()
          /home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/migrations_test.go:33 +0x52
      testing.tRunner()
          /home/travis/.gimme/versions/go/src/testing/testing.go:993 +0x1eb
    
    Previous write at 0x00c0000d2a68 by goroutine 77:
      github.com/hortbot/hortbot/internal/db/migrations/esc._escStaticFS.prepare.func1()
          /home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/esc/esc.go:59 +0xad
      sync.(*Once).doSlow()
          /home/travis/.gimme/versions/go/src/sync/once.go:66 +0x103
      sync.(*Once).Do()
          /home/travis/.gimme/versions/go/src/sync/once.go:57 +0x68
      github.com/hortbot/hortbot/internal/db/migrations/esc._escStaticFS.prepare()
          /home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/esc/esc.go:58 +0x160
      github.com/hortbot/hortbot/internal/db/migrations/esc._escStaticFS.Open()
          /home/travis/gopath/src/github.com/hortbot/hortbot/internal/db/migrations/esc/esc.go:78 +0x46
      github.com/hortbot/hortbot/internal/db/migrations/esc.(*_escStaticFS).Open()
          <autogenerated>:1 +0x5f
      github.com/golang-migrate/migrate/v4/source/httpfs.(*PartialDriver).ReadUp()
          /home/travis/gopath/pkg/mod/github.com/golang-migrate/migrate/[email protected]/source/httpfs/partial_driver.go:111 +0x1b9
      github.com/golang-migrate/migrate/v4/source/httpfs.(*driver).ReadUp()
          <autogenerated>:1 +0x5b
      github.com/golang-migrate/migrate/v4.(*Migrate).newMigration()
          /home/travis/gopath/pkg/mod/github.com/golang-migrate/migrate/[email protected]/migrate.go:837 +0x96
      github.com/golang-migrate/migrate/v4.(*Migrate).readUp()
          /home/travis/gopath/pkg/mod/github.com/golang-migrate/migrate/[email protected]/migrate.go:612 +0x22b
    

    It appears as though there's a path where the static file is being returned before it has been prepared, but I can't see how at first glance.

  • Precompute files MD5 / SHA1 / ...

    Precompute files MD5 / SHA1 / ...

    It would be useful to have the hashes of each files pre-computed and accessible with according methods so that we can send strong HTTP Etag headers for caching.

  • Doc Suggestion: Add example of Handlers for serving individual files

    Doc Suggestion: Add example of Handlers for serving individual files

    First of all, thanks for creating this! It works well and the API is clean and simple.

    I used it to bundle wasm and js into an example app so that a server and all content is bundled into the app. Some of my content is generated dynamically, so delegating everything to

    http.FileServer(FS(false))
    

    didn't seem workable. I ended up writing handler functions like the following to allow setting proper content headers. Seems to work very well, though I don't know if that's the cleanest possible way to code it.

    // fsSendStatic serves the requested file from the ESC FS.
    func fsSendStatic(w http.ResponseWriter, r *http.Request) {
    	_, err := w.Write(FSMustByte(false, r.URL.Path))
    	if err != nil {
    		log.Fatalf("Writing file %s returned error: %v", r.URL.Path, err)
    	}
    }
    
    // fsSendStaticJS serves the requested javascript file from the ESC FS.
    func fsSendStaticJS(w http.ResponseWriter, r *http.Request) {
    	w.Header().Set("Content-Type", "application/javascript")
    	fsSendStatic(w, r)
    }
    
    // fsSendStaticWasm serves the requested WebAssembly file  from the ESC FS.
    func fsSendStaticWasm(w http.ResponseWriter, r *http.Request) {
    	w.Header().Set("Content-Type", "application/wasm")
    	fsSendStatic(w, r)
    }
    

    It took a bit of trial and error to get it working. Just wondering if it might save others some time if the README included an example showing how to serve a mixture of static and dynamic content.

  • new esc.Run signature requires more template

    new esc.Run signature requires more template

    I recently upgraded to the newest version of this module and I noticed that esc.Run() now requires an io.Writer. That's awesome, however old code requires a lot of template (see https://github.com/StackExchange/dnscontrol/pull/515 for an example).

    Would it make sense to have (for example) a nil writer result in the file being created automatically? This would help with migrating old code, and also would reduce the amount of template code needed.

    Just a though!

A Go (golang) Custom Flutter Engine Embedder for desktop
A Go (golang) Custom Flutter Engine Embedder for desktop

Go Flutter desktop embedder ⚠️ Warning: this project has been moved to its own organization. Please take a look at its new location: github.com/go-flu

Jul 23, 2022
A zero dependency asset embedder for Go

Mewn A zero dependency asset embedder for Go. About Mewn is perhaps the easiest way to embed assets in a Go program. Here is an example: package main

Oct 23, 2022
This is a simple file storage server. User can upload file, delete file and list file on the server.
This is a simple file storage server.  User can upload file,  delete file and list file on the server.

Simple File Storage Server This is a simple file storage server. User can upload file, delete file and list file on the server. If you want to build a

Jan 19, 2022
A feature flag solution, with only a YAML file in the backend (S3, GitHub, HTTP, local file ...), no server to install, just add a file in a central system and refer to it. 🎛️
A feature flag solution, with only a YAML file in the backend (S3, GitHub, HTTP, local file ...), no server to install, just add a file in a central system and refer to it. 🎛️

??️ go-feature-flag A feature flag solution, with YAML file in the backend (S3, GitHub, HTTP, local file ...). No server to install, just add a file i

Dec 29, 2022
go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).
go-fastdfs 是一个简单的分布式文件系统(私有云存储),具有无中心、高性能,高可靠,免维护等优点,支持断点续传,分块上传,小文件合并,自动同步,自动修复。Go-fastdfs is a simple distributed file system (private cloud storage), with no center, high performance, high reliability, maintenance free and other advantages, support breakpoint continuation, block upload, small file merge, automatic synchronization, automatic repair.(similar fastdfs).

中文 English 愿景:为用户提供最简单、可靠、高效的分布式文件系统。 go-fastdfs是一个基于http协议的分布式文件系统,它基于大道至简的设计理念,一切从简设计,使得它的运维及扩展变得更加简单,它具有高性能、高可靠、无中心、免维护等优点。 大家担心的是这么简单的文件系统,靠不靠谱,可不

Jan 8, 2023
Simple Golang API to demonstrate file upload to fireabase storage and retrieving url of uploaded file.

go-firebase-storage -Work in progress ??️ Simple Golang API that uses Firebase as its backend to demonstrate various firebase services using Go such a

Oct 4, 2021
Simple-read-file - Example of how to read file in Go

simple-read-file This repository contains a simple example of how to read file i

Jan 11, 2022
Sqlyog-password-decoder - Simple decode passwords from .sycs file (SQLyog export connections file)

Decode password: ./sqlyog-password-decoder -str password -action decode Encode p

Nov 21, 2021
Convert scanned image PDF file to text annotated PDF file
Convert scanned image PDF file to text annotated PDF file

Jisui (自炊) This tool is PoC (Proof of Concept). Jisui is a helper tool to create e-book. Ordinary the scanned book have not text information, so you c

Dec 11, 2022
Parse awesome-go README file and generate a new README file with repo info.

Awesome Go Extra All data are from awesome-go and GitHub API. Audio and Music Libraries for manipulating audio. Name Description Star Open Issues Crea

Aug 11, 2022
This is a Go Cli app that receives an string path to a log file, and based on it generates and prints in console an encoded polyline with the locations found in the log file.
This is a Go Cli app that receives an string path to a log file, and based on it generates  and prints in console an encoded polyline with the locations found in the log file.

GEOENCODE GO CLI APP DESCRIPTION This is a Go Cli app that receives an string path to a log file, and based on it generates and prints in console an e

Oct 1, 2021
This command line converts .webarchive file to resources embed .html file

webarchive-to-singlefile This command line converts Safari's .webarchive file to complete .html. Notice Only tested on MacOS. Google Chrome required.

Dec 30, 2022
Distributed File Store Application Consist of API Server to handle file operations and command line tool to do operations

Filestore Distributed File Store Application Consist of API Server to handle file operations and command line tool to do operations (store named binar

Nov 7, 2021
Sort the emails contained in a .csv file into a text file

Go convert csv to txt This snippet of code allows you to sort the emails contained in a .csv file into a text file.

Nov 23, 2021
CLI filters the contents of the csv file according to the filters from the another file.

filtercsv CLI filters the contents of the csv file according to the filters from the another file. Made to process big files by a lots of filters. By

Dec 2, 2021
This command line converts thuderbird's exported RSS .eml file to .html file

thunderbird-rss-html This command line tool converts .html to .epub with images fetching. Install > go get github.com/gonejack/thunderbird-rss-html Us

Dec 15, 2021
DeepCopy a portable app that allows you to copy all forms of specified file types from your entire file system of the computer

DeepCopy a portable app that allows you to copy all forms of specified file types from your entire file system of the computer

Dec 20, 2021
Go-file-downloader-ftctl - A file downloader cli built using golang. Makes use of cobra for building the cli and go concurrent feature to download files.

ftctl This is a file downloader cli written in Golang which uses the concurrent feature of go to download files. The cli is built using cobra. How to

Jan 2, 2022
File-generator - Creates a file filled with bytes

creates a file filled with bytes ./filegen from to from : index of first piece t

Feb 14, 2022