An Experimental Wasm Virtual Machine for Gophers

gasm

CircleCI MIT License

A minimal implementation of v1 WASM spec compatible virtual machine purely written in go. The vm can be embedded in your go program without any dependency like cgo, and enables Gophers to write wasm host environments easily.

The vm should be used only for providing sandbox environments embedded in your Go program since we have not implemented validation of wasm binary.

The implementation is quite straightforward and I hope this code would be a good starting point for novices to learn WASM spec.

examples

Full examples can be found at: https://github.com/mathetake/gasm/tree/master/examples

call exported function from host

func Test_fibonacci(t *testing.T) {
	buf, _ := ioutil.ReadFile("wasm/fibonacci.wasm")
	mod, _ := wasm.DecodeModule(bytes.NewBuffer(buf))
	vm, _ := wasm.NewVM(mod, wasi.Modules)

	for _, c := range []struct {
		in, exp int32
	}{
		{in: 20, exp: 6765},
		{in: 10, exp: 55},
		{in: 5, exp: 5},
	} {
		ret, _, _ := vm.ExecExportedFunction("fib", uint64(c.in))
		require.Equal(t, c.exp, int32(ret[0]))
	}
}

call host function from WASM module

func Test_hostFunc(t *testing.T) {
	buf, _ := ioutil.ReadFile("wasm/host_func.wasm")
	mod, _ := wasm.DecodeModule(bytes.NewBuffer(buf))

	var cnt uint64  // to be incremented as hostFunc is called

	// host functions must be defined in the form of `Virtual Machine closure` generators
	// in order to access the VM state to get things done
	hostFunc := func(*wasm.VirtualMachine) reflect.Value {
		return reflect.ValueOf(func() {
			cnt++
		})
	}

	builder := hostfunc.NewModuleBuilderWith(wasi.Modules)
	builder.MustSetFunction("env", "host_func", hostFunc)
	vm, _ := wasm.NewVM(mod, builder.Done())

	for _, exp := range []uint64{5, 10, 15} {
		vm.ExecExportedFunction("call_host_func", exp)
		require.Equal(t, exp, cnt)
		cnt = 0
	}
}

🚧 WASI support 🚧

WebAssembly System Interface(WASI) will partly be supported in wasi package. Currently only fd_write is implemented and you can see it works in examples/panic example where the WASM module causes panic and the host prints the message through fd_write ABI.

references

https://webassembly.github.io/spec/core/index.html

Owner
Comments
  • JIT: Exception 0xc0000005 0x8 0x0 0x0

    JIT: Exception 0xc0000005 0x8 0x0 0x0

    I'm testing some code with Wazero, and I notice one odd crash, that I'm not sure if it's either my code (which is Zig) or Wazero. That is strange from previous errors, because it's one "Exception 0xc0000005 0x8 0x0 0x0". It's not an panic or something from testing.Error(err).

    I'm using Windows/amd64.


    The log is:

     go test -tags wasi,km,zig -v -bench=. -benchmem -benchtime=5s -cpu 1
    === RUN   TestEncodeObjectAPI
    Exception 0xc0000005 0x8 0x0 0x0                                                                                                                                              
    PC=0x0                                                                                                                                                                        
                                                                                                                                                                                  
    github.com/tetratelabs/wazero/internal/wasm/jit.(*callEngine).execWasmFunction(0xc00074a240, {0x5dd008, 0xc00009e140}, 0xc000952e70, 0xc000952cf0)                            
            Z:/GOPATH/pkg/mod/github.com/tetratelabs/[email protected]/internal/wasm/jit/engine.go:699 +0x185 fp=0xc00095dbb8 sp=0xc00095dae0 pc=0x4d5305 
    github.com/tetratelabs/wazero/internal/wasm/jit.(*moduleEngine).Call(0xc000954030?, {0x5dd008, 0xc00009e140}, 0xc000952e70, 0xc000494dd0, {0xc000954030, 0x1, 0x1})           
            Z:/GOPATH/pkg/mod/github.com/tetratelabs/[email protected]/internal/wasm/jit/engine.go:582 +0x48f fp=0xc00095dcb8 sp=0xc00095dbb8 pc=0x4d4b2f 
    github.com/tetratelabs/wazero/internal/wasm.(*FunctionInstance).Call(0x53bca0?, {0x5dd008?, 0xc00009e140?}, {0xc000954030?, 0xc000022270?, 0x0?})                             
            Z:/GOPATH/pkg/mod/github.com/tetratelabs/[email protected]/internal/wasm/call_context.go:164 +0x63 fp=0xc00095dd08 sp=0xc00095dcb8 pc=0x4901c3
    benchmark%2ekarmem%2eorg.(*Wasm).Run(...)
            X:/karmem/benchmark/main_wasi_test.go:289
    benchmark%2ekarmem%2eorg.TestEncodeObjectAPI(0xc000061040)
            X:/karmem/benchmark/main_wasi_test.go:113 +0x1f6 fp=0xc00095df70 sp=0xc00095dd08 pc=0x50d556
    testing.tRunner(0xc000061040, 0x58dce8)
            C:/Program Files/Go/src/testing/testing.go:1439 +0x102 fp=0xc00095dfc0 sp=0xc00095df70 pc=0x3e3482
    testing.(*T).Run.func1()
            C:/Program Files/Go/src/testing/testing.go:1486 +0x2a fp=0xc00095dfe0 sp=0xc00095dfc0 pc=0x3e432a
    runtime.goexit()
            C:/Program Files/Go/src/runtime/asm_amd64.s:1571 +0x1 fp=0xc00095dfe8 sp=0xc00095dfe0 pc=0x3587a1
    created by testing.(*T).Run
            C:/Program Files/Go/src/testing/testing.go:1486 +0x35f
    
    goroutine 1 [chan receive]:
    testing.(*T).Run(0xc000060ea0, {0x57e4e0?, 0x35af53?}, 0x58dce8)
            C:/Program Files/Go/src/testing/testing.go:1487 +0x37a
    testing.runTests.func1(0xc000020090?)
            C:/Program Files/Go/src/testing/testing.go:1839 +0x6e
    testing.tRunner(0xc000060ea0, 0xc0000c3cd8)
            C:/Program Files/Go/src/testing/testing.go:1439 +0x102
    testing.runTests(0xc000094320?, {0x72cd40, 0x1, 0x1}, {0x26848cc0598?, 0x40?, 0x738f20?})
            C:/Program Files/Go/src/testing/testing.go:1837 +0x457
    testing.(*M).Run(0xc000094320)
            C:/Program Files/Go/src/testing/testing.go:1719 +0x5d9
    main.main()
            _testmain.go:57 +0x1aa
    rax     0x20
    rbx     0x110000
    rcx     0xc000494270
    rdi     0x1100000012
    rsi     0xc0002de820
    rbp     0xc00095dba8
    rsp     0xc00095dad8
    r8      0x23280
    r9      0x8
    r10     0x90
    r11     0x0
    r14     0xc000966600
    r15     0xc004a90000
    rip     0x0
    rflags  0x10202
    cs      0x33
    fs      0x53
    gs      0x2b
    exit status 2
    

    That issue is ONLY affecst JIT, so I think it's an Wazero issue. The biggest issue is that it crashes the entire program and it's not possible to recover.

    Change:

    wazero.NewRuntimeConfigJIT().WithFinishedFeatures()
    

    To:

    wazero.NewRuntimeConfigInterpreter().WithFinishedFeatures()
    

    Runs the tests without issue:

    === RUN   TestEncodeObjectAPI
    --- PASS: TestEncodeObjectAPI (17.07s)
    

    Also, it's ONLY affects WASM compiled with zig build install -Dtarget="wasm32-wasi" -Drelease-safe, so if you compile as debug zig build install -Dtarget="wasm32-wasi" it will run. A lot slower, but runs. The debug have more bounds checks and so on. So, maybe there's something in my code that is actually crashing the runtime, crashing at one unrecoverable point.


    I'll share the source-code soon.

  • gojs: fix memory out of bounds

    gojs: fix memory out of bounds

    When in a loop reading a lot of data, the interpreter passes, but the compiler fails after this change. I'd love a hand from @mathetake to figure out the latter!

  • wasi: adds fd_readdir

    wasi: adds fd_readdir

    This adds an implementation of fd_readdir for WASI, which ensures a very large directory is not kept in host memory until its directory is closed.

    Original implementation and test data are with thanks from @jerbob92

  • wasi: exposes CloseWithExitCode to correct implementation of proc_exit

    wasi: exposes CloseWithExitCode to correct implementation of proc_exit

    Before, when the WASI function "proc_exit" was invoked, not only did the module stay alive, but also only the calling goroutine would receive an error with the exit code.

    panic(wasi.ExitCode(exitCode))
    

    Now, this corrects the implementation to actually shutdown the module, and ensure any callers can see the exit code, not just the caller who invoked exit.

    _ = m.CloseWithExitCode(exitCode)
    

    The changes needed for this were numerous, but allow other implementations, such as AssemblyScript abort handlers, to close the same way:

    • Adds Module.CloseWithExitCode which allows propagating an exit code to any callers of exported functions.
    • Dispatches Module.Close to Module.CloseWithExitCode(0)
    • Replace wasi.ExitCode with sys.ExitError which is returned to any in-flight callers. This formalizes the concept regardless of WASI.
    • Ensures all "close-once" guards are tested on engines.
    • Reuses the numeric field used for the closed flag to store the exit_code atomically.
    • Weave in exit errors into existing test cases to reduce code heft.

    This also folds the partially maintained WASI RATIONALE into the top-level doc.

  • Is it possible to grow api.Memory programmatically?

    Is it possible to grow api.Memory programmatically?

    There are scenarios where we may grow shared memory programmatically:

    module, _ := v.runtime.InstantiateModuleFromCode(ctx, policy)
    module.Memory().Grow(1) // error: unresolved reference 'Grow'
    

    However, the Grow method is not defined on the api.Memory interface. It's only defined on the internal MemoryInstance struct:

    func (m *MemoryInstance) Grow(newPages uint32) (result uint32) {
    	// [...]
    }
    

    Is it intentionally designed like that, or there is another API to grow shared memory?

  • Add emscripten_notify_memory_growth import?

    Add emscripten_notify_memory_growth import?

    Is your feature request related to a problem? Please describe. Compiling using emscripten requires emscripten_notify_memory_growth, defined at https://emscripten.org/docs/api_reference/emscripten.h.html#abi-functions.

    This is one single function and that is useless for Wazero. That is require for malloc.

    Minimal code:

    # include "stdint.h"
    # include "stdlib.h"
    # include "stdio.h"
    
    uint8_t *InputMemory;
    uint8_t *OutputMemory;
    
    int main() {
        OutputMemory = (uint8_t *) malloc(8000000);
        InputMemory = (uint8_t *) malloc(8000000);
    
        if (InputMemory == NULL || OutputMemory == NULL) {
            printf("Erro");
            return 1;
        } else {
            printf("Okay");
            return 0;
        }
    }
    

    Compile with:

    emcc wasm.c -o wasm.wasm -O3 -s ALLOW_MEMORY_GROWTH
    

    Requires emscripten_notify_memory_growth.

    Describe the solution you'd like Maybe Wazero can define such import, so any emscripten compiled wasm will work without declaring additional modules.

    Describe alternatives you've considered It's possible to fix by using:

        env, _ := host.NewModuleBuilder("env").
            ExportFunction("emscripten_notify_memory_growth", func(_ int32) {}).
            Instantiate(ctx)
    

    That is very small footprint, but very annoying to declare it everytime.

    Additional context Maybe we can do something similar of AssemblyScript and create one new Emscripten package? However, it's very small, maybe can be declared on WASI? But, it's not WASI anyway. :\

  • Add support for sign-extension instructions (post MVP feature)

    Add support for sign-extension instructions (post MVP feature)

    Hello,

    I'm opening this issue to report an error I ran into where it appeared wazero was unable to instantiate an AssemblyScript program compiled with asc.

    Here is the error I got:

    functions: invalid function at index 106/280: invalid instruction
    

    Tools like wasm2wat are able to read through the file without issues, so it appears correct tho I do not know whether they perform checks similar to those in wazero.

    I'm attaching the program that triggered the error: test.wasm.gz

    Let me know if you need any other information.

  • Published version v1.0.0-beta1 takes precedence over v1.0.0-beta.2

    Published version v1.0.0-beta1 takes precedence over v1.0.0-beta.2

    I don't know what caused this, since I see no tag v1.0.0-beta1, but according to semver v1.0.0-beta1 (notice the missing dot) will always take precedence over anything after a dot (like v1.0.0-beta.2).

    https://pkg.go.dev/github.com/tetratelabs/wazero?tab=versions

  • Custom data that is passed to host calls via `HostFunctionCallContext`

    Custom data that is passed to host calls via `HostFunctionCallContext`

    Hello! I am excited about wazero as a means to use Wasm without requiring CGO. Looking forward to ARM64 support!

    I was trying to add wazero as an engine to wapc-go and ran into a small snag. I need the ability to attach custom data to the module instance in order to store function invocation state (context, payload, error, etc). This option is available in other Wasm runtimes and should be easy to add to wazero.

    Envisioned usage:

    Instantiation

    	if err := m.store.Instantiate(m.module, moduleName); err != nil {
    		return nil, err
    	}
    
    	ic := invokeContext{
    		ctx: context.Background(),
    		// ctx and request payload is set prior to calling `store.CallFunction`
                    // response payload or error is set after calling the `hostCallHandler` (below)
    	}
    
    	m.store.SetInstanceData(moduleName, &ic)
    

    Invocation

    func (i *Instance) Invoke(ctx context.Context, operation string, payload []byte) ([]byte, error) {
    	*i.ic = invokeContext{
    		ctx:       ctx,
    		operation: operation,
    		guestReq:  payload,
    	}
    
    	results, _, err := i.m.store.CallFunction(i.name, "__guest_call", uint64(len(operation)), uint64(len(payload)))
    
    	// Inspect response payload or error in `invokeContext`
            // results[0] = 1 for success, 0 for failures
    }
    

    Host call

    func (m *Module) my_host_call(ctx *wasm.HostFunctionCallContext, operationPtr, operationLen, payloadPtr, payloadLen int32) int32 {
    	ic := ctx.Data.(*invokeContext)  // <--- Grabs the custom user data
    	data := ctx.Memory.Buffer
    	operation := string(data[operationPtr : operationPtr+operationLen])
    	payload := make([]byte, payloadLen)
    	copy(payload, data[payloadPtr:payloadPtr+payloadLen])
    
    	ic.hostResp, ic.hostErr = m.hostCallHandler(ic.ctx, operation, payload)
    	if ic.hostErr != nil {
    		return 0
    	}
    
    	return 1
    }
    

    Response payload or error are accessed via other host calls: full example here

  • Support WASI poll_oneoff for clock events

    Support WASI poll_oneoff for clock events

    Is your feature request related to a problem? Please describe. Related to https://github.com/tetratelabs/wazero/issues/271. The following code in Rust causes the following panic.

    std::thread::sleep(time::Duration::from_secs(10));
    
    thread '<unnamed>' panicked at 'thread::sleep(): unexpected result of poll_oneoff', library/std/src/sys/wasi/thread.rs:57:22
    

    Describe the solution you'd like Support for the poll_oneoff WASI method.

  • Allow passing fs.FS when calling functions

    Allow passing fs.FS when calling functions

    Issue

    Close https://github.com/tetratelabs/wazero/issues/563

    ToDo

    • [x] FSContext
    • [x] FSConfig
    • [x] FSKey
    • [x] Override fs.FS in WASI
    • [x] Add an example
    • [x] Unit test
    • [x] Example test
  • flakey test: gojs Test_stdio_large

    flakey test: gojs Test_stdio_large

    --- FAIL: Test_stdio_large (0.50s)
    [34](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:35)
        require.go:312: expected "stderr 2097152
    [35](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:36)
            ", but was "fatal error: malloc deadlock
    [36](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:37)
            
    [37](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:38)
            runtime stack:
    [38](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:39)
            runtime.throw({0x6b9ce, 0xf})
    [39](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:40)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/panic.go:1198 +0x7
    [40](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:41)
            runtime.mallocgc(0x10, 0x42cc0, 0x1)
    [41](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:42)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/malloc.go:972 +0xcf
    [42](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:43)
            runtime.newobject(0x42cc0)
    [43](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:44)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/malloc.go:1234 +0x4
    [44](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:45)
            runtime.handleEvent()
    [45](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:46)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/lock_js.go:238 +0x2
    [46](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:47)
            runtime.sysReserve(0x0, 0x800000)
    [47](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:48)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/mem_js.go:68 +0xe
    [48](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:49)
            runtime.sysReserveAligned(0x0, 0x400000, 0x400000)
    [49](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:50)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/malloc.go:805 +0x2
    [50](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:51)
            runtime.(*mheap).sysAlloc(0x329700, 0x400000)
    [51](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:52)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/malloc.go:698 +0x4f
    [52](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:53)
            runtime.(*mheap).grow(0x329700, 0x122)
    [53](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:54)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/mheap.go:1347 +0xa
    [54](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:55)
            runtime.(*mheap).allocSpan(0x329700, 0x122, 0x0, 0x1)
    [55](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:56)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/mheap.go:1179 +0x3b
    [56](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:57)
            runtime.(*mheap).alloc.func1()
    [57](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:58)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/mheap.go:913 +0x6
    [58](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:59)
            runtime.systemstack()
    [59](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:60)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/asm_wasm.s:170 +0x2
    [60](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:61)
            runtime.mstart()
    [61](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:62)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/asm_wasm.s:28
    [62](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:63)
            
    [63](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:64)
            goroutine 1 [running]:
    [64](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:65)
            runtime.systemstack_switch()
    [65](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:66)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/asm_wasm.s:181 fp=0x42ace8 sp=0x42ace0 pc=0x13ac0000
    [66](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:67)
            runtime.(*mheap).alloc(0x329700, 0x122, 0x1, 0x0)
    [67](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:68)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/mheap.go:907 +0x2 fp=0x42ad30 sp=0x42ace8 pc=0x11750002
    [68](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:69)
            runtime.(*mcache).allocLarge(0x350108, 0x244000, 0x0, 0x1)
    [69](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:70)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/mcache.go:227 +0x9 fp=0x42ad90 sp=0x42ad30 pc=0x10ed0009
    [70](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:71)
            runtime.mallocgc(0x244000, 0x0, 0x0)
    [71](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:72)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/malloc.go:1088 +0x8f fp=0x42ae18 sp=0x42ad90 pc=0x10a4008f
    [72](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:73)
            runtime.growslice(0x274c0, {0x500000, 0x1d0000, 0x1d0000}, 0x1d0001)
    [73](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:74)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/slice.go:261 +0x83 fp=0x42ae68 sp=0x42ae18 pc=0x12c60083
    [74](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:75)
            io.ReadAll({0xba820, 0x40c018})
    [75](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:76)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/io/io.go:631 +0x7 fp=0x42aee0 sp=0x42ae68 pc=0x15530007
    [76](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:77)
            github.com/tetratelabs/wazero/internal/gojs/testdata/stdio.Main()
    [77](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:78)
            	/home/runner/work/wazero/wazero/internal/gojs/testdata/stdio/main.go:10 +0x2 fp=0x42af28 sp=0x42aee0 pc=0x1f2f0002
    [78](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:79)
            main.main()
    [79](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:80)
            	/home/runner/work/wazero/wazero/internal/gojs/testdata/main.go:40 +0x2a fp=0x42af88 sp=0x42af28 pc=0x1f34002a
    [80](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:81)
            runtime.main()
    [81](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:82)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/proc.go:255 +0x35 fp=0x42afe0 sp=0x42af88 pc=0x121f0035
    [82](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:83)
            runtime.goexit()
    [83](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:84)
            	/opt/hostedtoolcache/go/1.17.13/x64/src/runtime/asm_wasm.s:431 +0x1 fp=0x42afe8 sp=0x42afe0 pc=0x13d50001
    [84](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:85)
            stderr 2097152
    [85](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:86)
            "
    [86](https://github.com/tetratelabs/wazero/actions/runs/3811321496/jobs/6483930063#step:4:87)
            /home/runner/work/wazero/wazero/internal/gojs/misc_test.go:56
    
  • Document best practices around invoking a wasi module multiple times

    Document best practices around invoking a wasi module multiple times

    Is your feature request related to a problem? Please describe. Not a problem per se, just questions around writing production-grade code with Wazero.

    Context: I have a simple Rust program that looks like this:

    use prql_compiler::compile;
    use std::io::{self, Read};
    
    fn main() {
        // example input is "from employees | select [name,age]  ";
    
        let mut prql = String::new();
        let stdin = io::stdin();
        let mut handle = stdin.lock();
        handle.read_to_string(&mut prql).expect("failed to read input"); // read stdin input
        let sql = compile(&prql).expect("failed to compile"); // transform the input into something else
        println!("{}", sql);
    }
    

    It's been compiled with cargo wasi build --release and produced a binary. Validating it works as expected with wasmtime:

    cat input.prql | wasmtime run target/wasm32-wasi/release/prql-wasi.wasm
    SELECT
      name,
      age
    FROM
      employees
    

    I would like to provide a Go library that embeds this wasm binary. Here's what it currently looks like, with non critical sections elided:

    //go:embed prql-wasi.wasm
    var prqlWasm []byte
    
    // Engine holds a reference to the wazero runtime as well as the pre-compiled wasm module
    type Engine struct {
    	code wazero.CompiledModule
    	r    wazero.Runtime
    }
    
    func (e *Engine) Compile(ctx context.Context, inBuf io.Reader, name string) (string, error) {
    	outBuf := new(bytes.Buffer)
    	errBuf := new(bytes.Buffer)
    	config := wazero.NewModuleConfig().
    		WithStdout(outBuf).WithStderr(errBuf).WithStdin(inBuf)
    
    	mod, err := e.r.InstantiateModule(ctx, e.code, config.WithName(name))
    	if err != nil {
    		if exitErr, ok := err.(*sys.ExitError); ok && exitErr.ExitCode() != 0 {
    			return "", err
    		} else if !ok {
    			return "", &ParseError{Input: "who knows", Err: err}
    		}
    	}
    	mod.Close(ctx)
    	errStream := errBuf.String()
    	if errStream != "" {
    		return "", fmt.Errorf(errStream)
    	}
    	return outBuf.String(), nil
    }
    
    func (e *Engine) Close(ctx context.Context) {
    	e.r.Close(ctx)
    }
    
    func New(ctx context.Context) *Engine {
    
    	r := wazero.NewRuntimeWithConfig(ctx, wazero.NewRuntimeConfig())
    	wasi_snapshot_preview1.MustInstantiate(ctx, r)
    	code, err := r.CompileModule(ctx, prqlWasm)
    	if err != nil {
    		log.Panicln(err)
    	}
    
    	return &Engine{
    		r:    r,
    		code: code,
    	}
    }
    

    which would be used like this:

    func main() {
    	ctx := context.Background()
    	p := prql.New(ctx)
    	defer p.Close(ctx)
    
    	content, _ := io.ReadAll(os.Stdin)
    	s := string(content)
    	inBuf := strings.NewReader(s)
           // simulate executing a transformation multiple times
    	for i := 0; i < 500; i++ {
                   name := fmt.Sprintf("run-%d", i)
    		_, err := p.Compile(ctx, inBuf,  name)
                    inBuf.Seek(0, 0)
                    // handle err…
           }
    

    Describe the solution you'd like

    It would be greatly beneficial to have documented best practices around how to optimize running a wasi module multiple times.

    Questions that could benefit from being documented

    What are the best practices around invoking a wasi module multiple times? What are the performance implications of using wasi vs a standard wasm module?

    Is it preferable to close the module each time:

    mod, _ := e.r.InstantiateModule(ctx, e.code, config)
    mod.Close()
    

    or invoking it with a different name without closing it:

    _, err := e.r.InstantiateModule(ctx, e.code, config.WithName(name))
    // is mod.Close() necessary if we use a different name for each invocation?
    

    Additional context Wazero has been extremely fun to build prototypes with, and I'm curious about how to start working on production-grade code. A canonical production ready example would be a great starting point for many new projects.

  • cache: crash on tinygo os package tests.

    cache: crash on tinygo os package tests.

    Describe the bug

    tinygo tests don't currently pass, but if you run with a cachedir on the second run, it panics like this

    $ wazero run -cachedir=$HOME/.wazero -mount=.:/ -env=HOME=.  os.wasm -test.v
    panic: runtime error: nil pointer dereference
    error instantiating wasm binary: module[] function[_start] failed: wasm error: unreachable
    wasm stack trace:
    	.runtime.runtimePanic(i32,i32)
    	.runtime.nilPanic()
    	.(*os.File).Write(i32,i32,i32,i32,i32)
    	.(*os.File).WriteString(i32,i32,i32,i32)
    	.os_test.writeFile(i32,i32,i32,i32,i32,i32)
    	.os_test.TestFstat(i32,i32)
    	.testing.tRunner(i32,i32,i32)
    	.testing.runTests$1(i32,i32)
    	.testing.tRunner(i32,i32,i32)
    	.(*testing.M).Run(i32) i32
    	.runtime.run$1()
    	.runtime.run$1$gowrapper(i32)
    	.tinygo_launch(i32)
    	._start()
    

    Note: I tried simpler wasm and it doesn't panic.

    To Reproduce

    either run from os.wasm.gz or build from a tinygo checkout

    ./build/tinygo test -target wasi -c -o os.wasm os
    

    Expected behavior should have same output

    Screenshots

    If I get the stacktrace from above, it looks like this:

    runtime/debug.Stack()
    	/usr/local/go/src/runtime/debug/stack.go:24 +0x65
    runtime/debug.PrintStack()
    	/usr/local/go/src/runtime/debug/stack.go:16 +0x19
    github.com/tetratelabs/wazero/internal/wasmdebug.(*stackTrace).FromRecovered(0xc0001896b8, {0x124dbe0?, 0xc0000b2390?})
    	/Users/adrian/oss/wazero/internal/wasmdebug/debug.go:1
    17 +0x3c
    github.com/tetratelabs/wazero/internal/engine/compiler.(*callEngine).deferredOnCall(0xc000125200, {0x124dbe0, 0xc0000b2390})
    	/Users/adrian/oss/wazero/internal/engine/compiler/engine.go:758 +0x2ed
    github.com/tetratelabs/wazero/internal/engine/compiler.(*callEngine).Call.func1()
    	/Users/adrian/oss/wazero/internal/engine/compiler/engine.go:650 +0x4d
    panic({0x124dbe0, 0xc0000b2390})
    	/usr/local/go/src/runtime/panic.go:884 +0x212
    github.com/tetratelabs/wazero/internal/engine/compiler.nativeCallStatusCode.causePanic(...)
    	/Users/adrian/oss/wazero/internal/engine/compiler/engine.go:427
    github.com/tetratelabs/wazero/internal/engine/compiler.(*callEngine).execWasmFunction(0xc000125200, {0x13026b8?, 0xc0000b7950?}, 0xc00072cc80)
    	/Users/adrian/oss/wazero/internal/engine/compiler/engine.go:944 +0x7a5
    github.com/tetratelabs/wazero/internal/engine/compiler.(*callEngine).Call(0xc000125200, {0x13026b8?, 0xc0000b7950?}, 0xc0000b91e0?, {0x0?, 0x6?, 0x0?})
    	/Users/adrian/oss/wazero/internal/engine/compiler/engine.go:659 +0x22c
    github.com/tetratelabs/wazero/internal/wasm.(*function).Call(0xc00072cc80?, {0x13026b8?, 0xc0000b7950?}, {0x0?, 0xc0000931e0?, 0x0?})
    	/Users/adrian/oss/wazero/internal/wasm/call_context.go:182 +0x48
    github.com/tetratelabs/wazero.(*namespace).InstantiateModule(0xc0000b39b0, {0x13026b8, 0xc0000b7950}, {0x1302aa0?, 0xc000742420}, {0x13042d0?, 0xc0000fe8c0})
    	/Users/adrian/oss/wazero/namespace.go:102 +0x2d9
    github.com/tetratelabs/wazero.(*runtime).InstantiateModule(0x13026b8?, {0x13026b8?, 0xc0000b7950?}, {0x1302aa0?, 0xc000742420?}, {0x13042d0?, 0xc0000fe8c0?})
    	/Users/adrian/oss/wazero/runtime.go:239 +0x3b
    main.doRun({0xc0000cc020, 0x5, 0x5}, {0x13014a0, 0xc0000c6008}, {0x1302030?, 0xc0000c6010?}, 0x12c3390)
    	/Users/adrian/oss/wazero/cmd/wazero/wazero.go:231 +0xe62
    main.doMain({0x13014a0, 0xc0000c6008}, {0x1302030?, 0xc0000c6010?}, 0x12c3390)
    	/Users/adrian/oss/wazero/cmd/wazero/wazero.go:54 +0x28b
    main.main()
    	/Users/adrian/oss/wazero/cmd/wazero/wazero.go:26 +0x3c
    
  • Deadlock in compilation cache

    Deadlock in compilation cache

    Describe the bug Use of the compilation cache between wazero versions seems to be able to cause a deadlock when the cache is cleaned.

    To Reproduce

    $ go mod init example.com/wazero-dl
    go: creating new go.mod: module example.com/wazero-dl
    $ cat <<EOF > main.go
    package main
    
    import (
            "context"
            "log"
            "os"
            "time"
    
            "github.com/tetratelabs/wazero"
            "github.com/tetratelabs/wazero/experimental"
    )
    
    func main() {
            bin, err := os.ReadFile(os.Args[1])
            if err != nil {
                    log.Panicln(err)
            }
    
            ctx, err := experimental.WithCompilationCacheDirName(context.Background(), ".build")
            if err != nil {
                    log.Panicln(err)
            }
            r := wazero.NewRuntime(ctx)
            defer r.Close(ctx) // This closes everything this Runtime created.
    
            start := time.Now()
            if _, err = r.CompileModule(ctx, bin); err != nil {
                    log.Panicln(err)
            }
            log.Printf("CompileModule took %dms", time.Since(start).Milliseconds())
    }
    EOF
    $ go get github.com/tetratelabs/wazero@3f578ddac3897b1435fbb1fe7e807676f062e4b1
    go: added github.com/tetratelabs/wazero v1.0.0-pre.5.0.20221229073945-3f578ddac389
    $ echo '(module (func (export "main") (result i32) i32.const 42 return))' | wat2wasm - -o simple.wasm
    $ go run . simple.wasm
    2022/12/29 15:16:56 CompileModule took 0ms
    $ go get github.com/tetratelabs/wazero@62be68dfc5f477d766d9dcfe422dcfc498c7bbfd
    go: downloading github.com/tetratelabs/wazero v1.0.0-pre.5.0.20221229092947-62be68dfc5f4
    go: upgraded github.com/tetratelabs/wazero v1.0.0-pre.5.0.20221229073945-3f578ddac389 => v1.0.0-pre.5.0.20221229092947-62be68dfc5f4
    $ go run . simple.wasm
    fatal error: all goroutines are asleep - deadlock!
    
    goroutine 1 [semacquire]:
    sync.runtime_SemacquireMutex(0x104425620?, 0xa0?, 0x1400013db68?)
            /Users/robbert/.gimme/versions/go1.19.3.darwin.arm64/src/runtime/sema.go:77 +0x24
    sync.(*RWMutex).Lock(0x101400013dba8?)
            /Users/robbert/.gimme/versions/go1.19.3.darwin.arm64/src/sync/rwmutex.go:152 +0x100
    github.com/tetratelabs/wazero/internal/compilationcache.(*fileCache).Delete(0x14000115170, {0xc1, 0x83, 0xb6, 0x11, 0xda, 0x74, 0xfe, 0xc0, 0x46, ...})
            /Users/robbert/go/pkg/mod/github.com/tetratelabs/[email protected]/internal/compilationcache/file_cache.go:92 +0x3c
    github.com/tetratelabs/wazero/internal/engine/compiler.(*engine).getCodesFromCache(0x1400010c0a0, 0x14000102680)
            /Users/robbert/go/pkg/mod/github.com/tetratelabs/[email protected]/internal/engine/compiler/engine_cache.go:84 +0x18c
    github.com/tetratelabs/wazero/internal/engine/compiler.(*engine).getCodes(0x1400013dd58?, 0x1043312f4?)
            /Users/robbert/go/pkg/mod/github.com/tetratelabs/[email protected]/internal/engine/compiler/engine_cache.go:35 +0x34
    github.com/tetratelabs/wazero/internal/engine/compiler.(*engine).CompileModule(0x1400010c0a0, {0x1044482b8, 0x14000115110}, 0x14000102680, {0x0, 0x0, 0x200?})
            /Users/robbert/go/pkg/mod/github.com/tetratelabs/[email protected]/internal/engine/compiler/engine.go:488 +0x3c
    github.com/tetratelabs/wazero.(*runtime).CompileModule(0x140001260c0, {0x1044482b8, 0x14000115110}, {0x14000164000, 0x26, 0x200})
            /Users/robbert/go/pkg/mod/github.com/tetratelabs/[email protected]/runtime.go:200 +0x258
    main.main()
            $/main.go:27 +0x160
    exit status 2
    

    Expected behavior No deadlock.

    Screenshots N/a

    Environment (please complete the relevant information):

    • Go version: 1.19.3
    • wazero Version: see repro, experienced on other commits too (se below).
    • Host architecture: arm64
    • Runtime mode: n/a

    Additional context Narrowed this down to a simple example from what I'd previously mentioned in https://github.com/tetratelabs/wazero/issues/939#issuecomment-1358589231. I could reproduce it it for the sha's mentioned there too but for the repo I used HEAD & HEAD~ to make sure this was still an issue.

  • [Feature Request] Support tracking which

    [Feature Request] Support tracking which "memory pages" are dirty

    Is your feature request related to a problem? Please describe. I'm exploring the possibility of "long running" WASM modules running in a distributed environment. To accomplish this, I'd like to be able to ensure durability of arbitrary WASM programs. I can use the existing Memory interface to "snapshot" the entire memory of the module after every invocation, but that's very expensive. I could log all function invocation requests, but that is also potentially very expensive for some workloads.

    Describe the solution you'd like I'd like to be able to track which "memory pages" have been dirtied since the previous invocation. I've created an experimental POC #968 where all store operations modify a bitset in the host memory indicating that a specific "page" of the WASM module memory has been dirtied for ARM architecture. The P.R is kind of a mess right now, but I think it demonstrates the basic idea. Currently it does this by default, but obviously I would make this an "opt in" feature.

    I think what I'm looking for here is the answer to a few questions:

    1. Does this feature interest you at all? If I was able to implement it for ARM and x86 + make it opt-in so it has 0 cost for workloads that don't use it, and also expose it via some reasonable APIs for user who do opt in to it, would you consider merging it?
    2. Does my implementation make any sense? Is there a better way to accomplish what I want?

    Describe alternatives you've considered I don't really have any other ideas except the expensive ways I described in the first paragraph.

    Additional context N/A

  • [Draft] [Experimental] Add support for tracking dirty memory pages

    [Draft] [Experimental] Add support for tracking dirty memory pages

    P.O.C for #969

    P.R is touching many files just to wire a new options struct through the APIs, but the main changes are in:

    1. internal/engine/compiler/engine.go
    2. internal/engine/compiler/impl_arm64.go

    TODOs:

    1. Get some feedback on the assembly implementation since I'm terrible at writing assembly code.
    2. Modify builtinFunctionMemoryGrow to increase the size of the bitset based on the tracking size and don't hardcode the size of the []byte backing the bitset to such a large value. Note that technically its legal for the host to grow the memory size manually which means that we may need to add some kind of check before each invocation to see that the bitset size is large enough to accomodate the memory size.
    3. Clear the bitset between invocations.
    4. Add more unit tests to ensure the right bits are actually set, the bitset is cleared between invocations, and that the bitset grows as the memory grows automatically. Note that it may make more sense to move the bitset to the memory instance object directly instead of its own sub struct.
    5. Modify some of the existing integration tests to run twice, once with TrackDirtyMemoryPages set to true for increased coverage.
    6. Add APIs on the Memory instance so the host can actually inspect the dirty pages after every invocation.
DOM library for Go and WASM

Go DOM binding (and more) for WebAssembly This library provides a Go API for different Web APIs for WebAssembly target. It's in an active development,

Dec 23, 2022
Library to use HTML5 Canvas from Go-WASM, with all drawing within go code

go-canvas go-canvas is a pure go+webassembly Library for efficiently drawing on a html5 canvas element within the browser from go without requiring ca

Dec 11, 2022
Run WASM tests inside your browser

wasmbrowsertest Run Go wasm tests easily in your browser. If you have a codebase targeting the wasm platform, chances are you would want to test your

Dec 16, 2022
Golang-WASM provides a simple idiomatic, and comprehensive API and bindings for working with WebAssembly for Go and JavaScript developers
Golang-WASM provides a simple idiomatic, and comprehensive API and bindings for working with WebAssembly for Go and JavaScript developers

A bridge and bindings for JS DOM API with Go WebAssembly. Written by Team Ortix - Hamza Ali and Chan Wen Xu. GOOS=js GOARCH=wasm go get -u github.com/

Dec 22, 2022
Go Wasm is a in-browser IDE for Go

Go Wasm Go Wasm is a Go development environment with the essentials to write and run code entirely within the browser, using the power of WebAssembly

Jan 8, 2023
A WASM Filter for Envoy Proxy written in Golang

envoy-proxy-wasm-filter-golang A WASM Filter for Envoy Proxy written in Golang Build tinygo build -o optimized.wasm -scheduler=none -target=wasi ./mai

Nov 6, 2022
Istio wasm api demo with golang

istio-wasm-api-demo 1. Setup the latest Istio Setup k8s cluster: e.g. kind create cluster --name test Download the latest Istioctl from the GitHub rel

Nov 1, 2022
Vugu: A modern UI library for Go+WebAssembly (experimental)

Vugu: A modern UI library for Go+WebAssembly (experimental)

Jan 3, 2023
Aes for go and java; build go fo wasm and use wasm parse java response.

aes_go_wasm_java aes for go and java; build go fo wasm and use wasm parse java response. vscode setting config settings.json { "go.toolsEnvVars":

Dec 14, 2021
Virtual-Operating-System - Virtual Operating System Using Golang And Fyne Implemented Gallery app
Virtual-Operating-System - Virtual Operating System Using Golang And Fyne Implemented Gallery app

Virtual Operating System Virtual Operating System Using Golang And Fyne Implemen

Jan 1, 2022
An embeddable implementation of the Ngaro Virtual Machine for Go programs

Ngaro Go Overview This is an embeddable Go implementation of the Ngaro Virtual Machine. This repository contains the embeddable virtual machine, a rud

Dec 3, 2022
A bytecode-based virtual machine to implement scripting/filtering support in your golang project.

eval-filter Implementation Scripting Facilities Types Built-In Functions Conditionals Loops Functions Case/Switch Use Cases Security Denial of service

Dec 30, 2022
Forth virtual machine in Go

forego - A Forth implementation in Go ===================================== Why? ---- For ego. This is me learning the language. Both of them. So

Sep 9, 2022
A customisable virtual machine written in Go

== About GoLightly == GoLightly is a lightweight virtual machine library implemented in Go, designed for flexibility and reuse. Traditionally popular

Nov 16, 2022
A simple virtual machine - compiler & interpreter - written in golang

go.vm Installation Build without Go Modules (Go before 1.11) Build with Go Modules (Go 1.11 or higher) Usage Opcodes Notes The compiler The interprete

Dec 17, 2022
A bytecode-based virtual machine to implement scripting/filtering support in your golang project.

eval-filter Implementation Scripting Facilities Types Built-In Functions Conditionals Loops Functions Case/Switch Use Cases Security Denial of service

Jan 8, 2023
Expr – a tiny stack-based virtual machine written in Go

Expr – a tiny stack-based virtual machine written in Go The executor is designed to interpret a simple expression language and it's useful in delegati

Nov 11, 2022
A Cloud Native Buildpack that contributes SDKMAN and uses it to install dependencies like the Java Virtual Machine

gcr.io/paketo-buildpacks/sdkman A Cloud Native Buildpack that contributes SDKMAN and uses it to install dependencies like the Java Virtual Machine. Be

Jan 8, 2022
Weave Ignite is an open source Virtual Machine (VM) manager with a container UX and built-in GitOps management.
Weave Ignite is an open source Virtual Machine (VM) manager with a container UX and built-in GitOps management.

Weave Ignite is an open source Virtual Machine (VM) manager with a container UX and built-in GitOps management.

Nov 16, 2021
Exploratory implementation of the Eva virtual machine

Eva Exploratory implementation of the Eva virtual machine in pure Go. Eva is a simple virtual machine designed for educational use. This is not intend

Dec 27, 2021