Unfancy resources embedding for Go with out of box http.FileSystem support.

Resources GoDoc Build Status Go Report Card

Unfancy resources embedding with Go.

  • No blings.
  • No runtime dependency.
  • Idiomatic Library First design.

Dude, Why?

Yes, there is quite a lot of projects that handles resource embedding but they come with more bling than you ever need and you often end up with having dependencies for your end project. Not this time.

Installing

Just go get it!

$ go get github.com/omeid/go-resources/cmd/resources

Usage

$ resources -h
Usage resources:
  -declare
        whether to declare the -var (default false)
  -fmt
        run output through gofmt, this is slow for huge files (default false)
  -output filename
        filename to write the output to
  -package name
        name of the package to generate (default "main")
  -tag tag
        tag to use for the generated package (default no tag)
  -trim prefix
        path prefix to remove from the resulting file path in the virtual filesystem
  -var name
        name of the variable to assign the virtual filesystem to (default "FS")
  -width number
        number of content bytes per line in generetated file (default 12)

Optimization

Generating resources result in a very high number of lines of code, 1MB of resources result about 5MB of code at over 87,000 lines of code. This is caused by the chosen representation of the file contents within the generated file.

Instead of a (binary) string, resources transforms each file into an actual byte slice. For example, a file with content Hello, world! will be represented as follows:

FS = &FileSystem{
  "/hello.txt": File{
    data: []byte{
      0x48, 0x65, 0x6c, 0x6c, 0x6f, 0x2c, 0x20, 0x57, 0x6f, 0x72, 0x6c, 0x64,
      0x21,
    },
    fi: FileInfo{
      name:    "hello.txt",
      size:    13,
      modTime: time.Unix(0, 1504640959536230658),
      isDir:   false,
    },
  },
}

While this seems wasteful, the compiled binary is not really affected. If you add 1MB of resources, your binary will increase 1MB as well.

However, compiling this many lines of code takes time and slows down the compiler. To avoid recompiling the resources every time and leverage the compiler cache, generate your resources into a standalone package and then import it, this will allow for faster iteration as you don't have to wait for the resources to be compiled with every change.

mkdir -p assets
resources -declare -var=FS -package=assets -output=assets/assets.go your/files/here
package main

import "importpath/to/assets"

func main() {
  data, err := assets.FS.Open("your/files/here")
  // ...
}
"Live" development of resources

For fast iteration and improvement of your resources, you can work around the compile with the following technique:

First, create a normal main.go:

package main

import "net/http"

var Assets http.FileSystem

func main() {
  if Assets == nil {
    panic("No Assets. Have you generated the resources?")
  }

  // use Assets here
}

Then, add a second file in the same package (main here), with the following content:

// +build !embed

package main

import (
	"net/http"

	"github.com/omeid/go-resources/live"
)

var Assets = live.Dir("./public")

Now when you build or run your project, you will have files directly served from ./public directory.

To create a production build, i.e. one with the embedded files, build the resouces with -tag=embed and add the embed tag to go build:

$ resources -output=public_resources.go -var=Assets -tag=embed public/*
$ go build -tags=embed

Now your resources should be embedded with your program! Of course, you may use any var or tag name you please.

Go Generate

There is a few reasons to avoid resource embedding in go generate.

First go generate is for generating Go source code from your code, generally the resources you want to embed aren't effected by the Go source directly and as such generating resources are slightly out of the scope of go generate.

Second, you're unnecessarily slowing down code iterations by blocking go generate for resource generation.

Resources, The Library GoDoc

The resource generator is written as a library and isn't bound to filesystem by the way of accepting files in the form

type File interface {
      io.Reader
      Stat() (os.FileInfo, error)
}

along with a helper method that adds files from filesystem.

This allows to integrate resources with ease in your workflow when the when the provided command doesn't fit well, for an example see the Gonzo binding resources.

Please refer to the GoDoc for complete documentation.

Strings

The generated FileSystem also implements an String(string) (string, bool) method that allows you to read the content of a file as string, to use that instead of defining your file Assets variable as simply an http.FileSystem, do the following:

type Resources interface {
	http.FileSystem
	String(string) (string, bool)
}

var Assets Resources

Now you can call Assets.String(someFile) and get the content as string with a boolean value indicating whatever the file was found or not.


Contributing

Please consider opening an issue first, or just send a pull request. :)

Credits

See Contributors.

LICENSE

MIT.

TODO

  • Add tests.
Owner
O'meid
dead heroes can't tame the wicked.
O'meid
Comments
  • Improving directory and mass file support

    Improving directory and mass file support

    Hey Omeid!

    I've been using your go-resources for quite a while and it served me well, so thank you so much for sharing it!

    Today I wanted to embed quite a few files (649 to be exact), which failed with "too many open files". Also I noticed while implementing Readdir[*] for #4 (which I needed as well) that go-resources kinda only has the root dir and all files are in this global map. Also directories are assumed when a file has a certain prefix, e.g. "/00" is recognised as a directory for "/001", "/002", "/003" and so on.

    So I thought I'd fix the "too many open files" error and change the implementation to represent actual directories instead of a plain map. These are the steps I was thinking about:

    • Pre-render files in AddFile, so we can close the file handle right away
    • FileSystem contains a root file instead of the files map
    • Change FileSystem.files to represent only the direct children of the root dir
      • directory nodes will contain their children
    • FileInfo will contain a []string for file names as well, so we have an indexed list for Readdir's count parameter
    • FileSystem.Open only returns a directory if it actually is one
    • Maybe gofmt the output (No idea how, it's probably easier to format the template)

    Thoughts?

    [*] I fucked that up btw, since I totally missed that files is a map and therefor can't be properly sliced.

  • Inconsistency between live directory and embedded fs paths

    Inconsistency between live directory and embedded fs paths

    At the moment, it's impossible to have a mirrored live directory and an embedded filesystem.

    Assume that you have the following directory layout:

    /
    | ---- resources/ 
              |--------- res_manager.go
              |--------- embedded.go
              |--------- non-embed.go
              |--------- web/
                          |----- mycss.css
                          |----- myimage.png
              |--------- snd/
                          |----- mysnd.ogg
                          |----- othersnd.ogg
    

    generating the embedded file system (embedded.go) via

    $GOPATH/bin/resources -tags embed -declare -package resources -output embedded.go web/* snd/*
    

    will result in the following path-mapping:

    /web/mycss.css
    /web/myimage.png
    /snd/mysnd.ogg
    /snd/othersnd.ogg   
    

    However, serving the live directory (non-embed.go) via:

       live.Dir("./")
    

    will result in the following path-mapping:

    web/mycss.css
    web/myimage.css
    snd/mysnd.ogg
    snd/othersnd.ogg
    

    (needless to say, using / as the path will map the entire filesystem, outside of a chroot, which is bad.)

    This, as a result, causes a disconnect between common code, as the mapping must be exact for code to actually find the file. The -trim argument, also fails to do this, because it is hard coded in the template.

    Tangentially related: #15

  • creates invalid code on AppVeyor/Windows

    creates invalid code on AppVeyor/Windows

    go get github.com/omeid/go-resources/cmd/resources
    resources -declare -var=DEFAULTS -package=assets -output=core/assets/assets.go core/config/*.json
    2017/09/13 13:41:50 Finished in 261.0261ms. Wrote 2 resources to core/assets/assets.go
    go test ./...
    # github.com/ethereumproject/go-ethereum/core/assets
    core\assets\assets.go:140:11: unknown escape sequence
    core\assets\assets.go:92927:6: duplicate key "" in map literal
    core\assets\assets.go:92927:13: unknown escape sequence
    

    See https://github.com/whilei/go-resources/commits/appveyable + https://ci.appveyor.com/project/whilei/go-resources for simple reproducible test.

  • Resources generates a broken Go code

    Resources generates a broken Go code

    Hi. I am new to Go and Go resources. It looks pretty good. However, I face an issue when using it. When I try to generate code from a simple json file, resources throws an error and produce a broken Go file.

    resources -output="resources.go" -var="Resources" resources/manifest.json                          
    2017/06/15 09:37:38 template: file:3:4: executing "file" at <reader .>: error calling reader: EOF 
    

    resources/manifest.json:

    {
      "name": "agent-ping"
    }
    

    I did some tests and the problem was introduced on commit 1efd65f. Before this, the code generation succeeded with the same command and inputs.

  • Can't readdir files in

    Can't readdir files in "/"

    Right now following code returns error (where fs is generated http.FileSystem implementation), and doesn't return list of files.

        d, err := fs.Open("/")
        if err != nil {
            return nil, err
        }
    
        files, err := d.Readdir(0)
        if err != nil {
            return nil, err
        }
    

    Here is a patch that fixes that:

    diff --git a/resources.go b/resources.go
    index daeccd7..4591958 100644
    --- a/resources.go
    +++ b/resources.go
    @@ -188,6 +188,10 @@ func (f *File) Close() error {
     }
    
     func (f *File) Readdir(count int) ([]os.FileInfo, error) {
    +  if f.fi.isDir {
    +    return f.fi.files, nil
    +  }
    +
       return nil, os.ErrNotExist
     }
    
    @@ -226,10 +230,6 @@ func (f *FileInfo) IsDir() bool {
            return f.isDir
     }
    
    -func (f *FileInfo) Readdir(count int) ([]os.FileInfo, error) {
    -       return f.files, nil
    -}
    -
     func (f *FileInfo) Sys() interface{} {
            return f.sys
     }
    
  • How to add files from subfolder to resource asset?

    How to add files from subfolder to resource asset?

    Hello!

    I'm trying to add files from subfolder to asset:

    ls -al resources/mongodb/repolist.static 
    -rw-r--r-- 1 root root 73 Jan 14 11:18 resources/mongodb/repolist.static
    

    But it's not working with following call:

    resources -output="public_resources.go" -var="StaticResources" resources/*
    

    Actually "mongodb" folder itself have added correctly:

    cat public_resources.go |grep mongodb -C 5
                }, "/resources/mongodb": File{
                    data: []byte{},
                    fi: FileInfo{
                        name:    "mongodb",
                        size:    4096,
                        modTime: time.Unix(1452847762, 1452847762787760103),
                        isDir:   true,
                    },
    
    

    But nested file "repolist.static" haven't added :(

    Do you have any plans to implement this feature?

  • Allow live.Dir to Work With Go Modules

    Allow live.Dir to Work With Go Modules

    Currently, any repo that uses Go Modules will get an error if they try to open a file from live.Dir:

    open github.com/chabad360/covey/assets/base.html: no such file or directory
    

    This is because live.Dir uses runtime.Caller which returns the module path to the caller. This PR uses os.Executable() to fix that.

    This PR fixes #18 (and doesn't require #19).

  • Forcing path to / results in weird defaults

    Forcing path to / results in weird defaults

    Running resources with relative paths like ../logo.png results in assets that have a weird name like "/../logo.png" (parent of the root?).

    I'd find it more predictable if the default asset names are identical to the paths used during generation. Perhaps an new option --root could be added to keep the current behavior?

    https://github.com/omeid/go-resources/blob/46f4269d8abdc8829f45595f162f6bd0b7ee2a16/resources.go#L299

  • invalid memory address or nil pointer dereference when accessing to npt existing path

    invalid memory address or nil pointer dereference when accessing to npt existing path

    Hello!

    Thanks for nice code!

    But I have found small mistake.

    When I trying to open resource which I haven't in my asset I got memory access issue:

    Trying to open not existent resource
    panic: runtime error: invalid memory address or nil pointer dereference
    [signal 0xb code=0x1 addr=0x8 pc=0x4ac2d1]
    
    goroutine 1 [running]:
    bytes.(*Reader).Read(0x0, 0xc8200a5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/bytes/reader.go:42 +0x61
    bufio.(*Reader).Read(0xc820018120, 0xc8200a5000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        /usr/local/go/src/bufio/bufio.go:197 +0x126
    bufio.(*Scanner).Scan(0xc82004fed0, 0x1000)
        /usr/local/go/src/bufio/scan.go:180 +0x877
    main.main()
        /root/fastnetmon/src/installer/selfbuild.go:39 +0x54b
    
    goroutine 17 [syscall, locked to thread]:
    runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1
    

    My code is:

    package main
    
    // Install:
    // go get github.com/omeid/go-resources/cmd/resources
    
    // Generate resources:
    // resources -output="public_resources.go" -var="StaticResources" resources/*
    // Build project:
    // go build
    
    import (
        "fmt"
        "net/http"
        "log"
        "bufio"
    )
    
    // All files from resources folder availible here
    var StaticResources http.FileSystem 
    
    func main() {
        if StaticResources == nil {
            log.Fatal("No StaticResources. Have you generated the resources?")   
        }
    
        fmt.Println("Trying to open not existent resource")
        static_resource, err := StaticResources.Open("/resources/nginx_gpg_signing_key.staticXXXX")
        if err != nil {
            log.Fatal("Could not open resource nginx_gpg_signing_key.static")
        }   
    
        defer static_resource.Close()
    
        reader := bufio.NewReader(static_resource)
        scanner := bufio.NewScanner(reader)
    
        for scanner.Scan() {
            fmt.Println(scanner.Text())
        }
    }
    
    
  • Readdir() returns invalid result – one file many times

    Readdir() returns invalid result – one file many times

    FileSystem has a map of File structures:

    type FileSystem struct {
        files map[string]File
    }
    

    When iterating over this map in FileSystem.Open function, each File value from map is copied into file variable:

            for path, file := range fs.files {
                if strings.HasPrefix(path, name) {
                    s, _ := file.Stat()
                    files = append(files, s)
                }
            }
    

    When file.Stat() is called, what actually happens is (&file).Stat() (because Stat() is only defined on *File, not File), and Stat() returns

    func (f *File) Stat() (os.FileInfo, error) {
      return &f.fi, nil
    }
    

    That is, address of fi field of file variable from for loop in our case. This is then added to the result.

    Subsequent iterations overwrite file variable, and thus also its fi field, but file.Stat() still returns the same address -- one that was used in previous iterations as well.

    End result is that files slice contains same result several times.

    0: *main.FileInfo 0xc82009c380
    1: *main.FileInfo 0xc82009c380
    2: *main.FileInfo 0xc82009c380
    

    We can fix that by doing

            for path, file := range fs.files {
                if strings.HasPrefix(path, name) {
                    f := file // copy to new variable, that will end up in the `files`
                    s, _ := f.Stat()
                    files = append(files, s)
                }
            }
    
  • Wrong FileInfo.modTime generation

    Wrong FileInfo.modTime generation

    Hi,

    I've noticed a small issue with the FileInfo.modTime generation. It is a bit annoying because it prevents proper client-side caching when serving the contents via HTTP.

    In the generated file, I find this entry:

    FileInfo{
    	modTime: time.Unix(1504547519, 1504547519143839891),
    	// etc.
    }
    

    However, time.Unix will return 2065-05-09T12:43:58+01:00, the correct value (ignoring time zones) is:

    $ stat -c %y ui/build/style.css
    2017-09-04 19:51:59.143839891 +0200
    

    While the documentation of time.Unix states

    It is valid to pass nsec outside the range [0, 999999999]

    it does not mention the implementation detail that it adds nsec / 1e9 to the sec value. Hence the generated FileInfo should be in either of these forms:

    time.Unix(0, 1504547519143839891) // 0, nsec
    time.Unix(1504547519, 143839891)  // nsec/1e9, nsec%1e9
    

    I believe the first form should work on any Go release (the crucial part in the implementation of Go1 and Go1.9 has stayed the same).

Takes an input http.FileSystem (likely at go generate time) and generates Go code that statically implements it.

vfsgen Package vfsgen takes an http.FileSystem (likely at go generate time) and generates Go code that statically implements the provided http.FileSys

Dec 18, 2022
go.rice is a Go package that makes working with resources such as html,js,css,images,templates, etc very easy.

go.rice go.rice is a Go package that makes working with resources such as html,js,css,images and templates easy. During development go.rice will load

Dec 29, 2022
Golog is a logger which support tracing and other custom behaviors out of the box. Blazing fast and simple to use.

GOLOG Golog is an opinionated Go logger with simple APIs and configurable behavior. Why another logger? Golog is designed to address mainly two issues

Oct 2, 2022
:file_folder: Embeds static resources into go files for single binary compilation + works with http.FileSystem + symlinks

Package statics Package statics embeds static files into your go applications. It provides helper methods and objects to retrieve embeded files and se

Sep 27, 2022
Open Service Mesh (OSM) is a lightweight, extensible, cloud native service mesh that allows users to uniformly manage, secure, and get out-of-the-box observability features for highly dynamic microservice environments.
Open Service Mesh (OSM) is a lightweight, extensible, cloud native service mesh that allows users to uniformly manage, secure, and get out-of-the-box observability features for highly dynamic microservice environments.

Open Service Mesh (OSM) Open Service Mesh (OSM) is a lightweight, extensible, Cloud Native service mesh that allows users to uniformly manage, secure,

Jan 2, 2023
⚗ The most advanced CLI template on earth! Featuring automatic releases, website generation and a custom CI-System out of the box.
⚗ The most advanced CLI template on earth! Featuring automatic releases, website generation and a custom CI-System out of the box.

cli-template ✨ ⚗ A template for beautiful, modern, cross-platform compatible CLI tools written with Go! Getting Started | Wiki This template features

Dec 4, 2022
Take control over your live stream video by running it yourself. Streaming + chat out of the box.
Take control over your live stream video by running it yourself.  Streaming + chat out of the box.

Take control over your content and stream it yourself. Explore the docs » View Demo · Use Our Server for Testing · FAQ · Report Bug Table of Contents

Jan 1, 2023
A minimal Go project with user authentication ready out of the box. All frontend assets should be less than 100 kB on every page load

Golang Base Project A minimal Golang project with user authentication ready out of the box. All frontend assets should be less than 100 kB on every pa

Jan 1, 2023
Golang-echo-sample - Make an out-of-the-box backend based on golang-echo

Golang-echo-sample - Make an out-of-the-box backend based on golang-echo

Dec 31, 2021
An out-of-the-box cryptographic message communication.

An out-of-the-box cryptographic message communication.

Feb 8, 2022
Issue-mafia - An out-of-the-box CLI that helps you to easily synchronize Git hooks with a remote repository

issue-mafia is an out-of-the-box CLI that helps you to easily synchronize Git hooks with a remote repository.

Feb 14, 2022
a lightweight, high-performance, out-of-the-box logging library that relies solely on the Go standard library

English | 中文 olog olog is a lightweight, high-performance, out-of-the-box logging library that relies solely on the Go standard library. Support outpu

Apr 12, 2023
In 'n Out - See what goes in and comes out of PEs/DLLs

In 'n Out Parse and return PE information ino -v comsvcs.dll { "Name": "<string>", "Path": "<string>", "Type": "<string file|directory>", "Im

Dec 16, 2022
This package is built for Embedding PHP into Golang.

GoEmPHP This package is built for Embedding PHP into Golang. It is easy to use: script = php.New() script.Startup() defer script.Close()

Jul 2, 2022
A distributed system for embedding-based retrieval
A distributed system for embedding-based retrieval

Overview Vearch is a scalable distributed system for efficient similarity search of deep learning vectors. Architecture Data Model space, documents, v

Dec 30, 2022
Flutter on Windows, MacOS and Linux - based on Flutter Embedding, Go and GLFW.
Flutter on Windows, MacOS and Linux - based on Flutter Embedding, Go and GLFW.

go-flutter - A package that brings Flutter to the desktop Purpose Flutter allows you to build beautiful native apps on iOS and Android from a single c

Jan 9, 2023
Example repository for embedding Litestream in a Go application.

Litestream as Library This repository is an example of embedding Litestream as a library in a Go application. The Litestream API is not stable so you

Dec 6, 2022
An open source embedding vector similarity search engine powered by Faiss, NMSLIB and Annoy
An open source embedding vector similarity search engine powered by Faiss, NMSLIB and Annoy

Click to take a quick look at our demos! Image search Chatbots Chemical structure search Milvus is an open-source vector database built to power AI ap

Jan 7, 2023
Arche - Smart Hybrid Workforce Manager: A system that aims to provide companies an easy to use platform for managing company resources by allowing employees to book company spaces and resources.
Arche - Smart Hybrid Workforce Manager: A system that aims to provide companies an easy to use platform for managing company resources by allowing employees to book company spaces and resources.

Description Smart Hybrid Workforce Manager is a system that aims to provide companies an easy to use system for managing company resources by allowing

Dec 8, 2022
🔥 Golang live stream lib/client/server. support RTMP/RTSP/HLS/HTTP[S]-FLV/HTTP-TS, H264/H265/AAC, relay, cluster, record, HTTP API/Notify, GOP cache. 官方文档见 https://pengrl.com/lal
🔥 Golang live stream lib/client/server. support RTMP/RTSP/HLS/HTTP[S]-FLV/HTTP-TS, H264/H265/AAC, relay, cluster, record, HTTP API/Notify, GOP cache. 官方文档见 https://pengrl.com/lal

lal是一个开源GoLang直播流媒体网络传输项目,包含三个主要组成部分: lalserver:流媒体转发服务器。类似于nginx-rtmp-module等应用,但支持更多的协议,提供更丰富的功能。lalserver简介 demo:一些小应用,比如推、拉流客户端,压测工具,流分析工具,调度示例程序等

Jan 1, 2023