Runwasi - A containerd shim which runs wasm workloads in wasmtime

This is a series of patches to add support for running wasm code in a cgroup. While some cgroup controllers can support running different threads in different cgroups, the main one we are interested in, the cgroup memory controller, this doesn't make sense since threads all share memory. Because of this we'll fork off a new process (which should have CoW version of the main shim's memory) to run the wasm code.

Of course all the issues of fork apply here since this is fairly unsafe from a multithreaded program (which the shim is always multi-threaded), so care must be taken to not try do things like take a lock in the new process because this can cause a deadlock. There may be some interesting things to test out with the wasmtime engine just to make sure we aren't going to deadlock if there's multiple things happening when the fork occurs.

People need to compile this project by using Rust in beta or nightly channel.

So I add a rust-toolchain.toml file to pin the version, so people can run cargo build command directly

Signed-off-by: Manjusaka [email protected]

Currently there is no troubleshotting guide. People might find it hard to follow readme to produce a hello world example.

Known issues are, but not limited to:

1. containerd currently only support Linux. So in order to build runwasi, either you need to have a linux machien or run it in WSL on Windows
2. docker buildx is a dependency
3. make load is broken

A cgroup can either have delegations (cgroup.subtree_control is populated) OR it can have processes in it, but not both. So we need to make sure that we don't add any controllers to the new cgroup's cgroup.subtree_control BUT we should continue to do so for any parents that we create.

Fixes #35

Not fail on every machines.

I have success and fail machines both, but I am not sure what kind of information I could provide. I could help test in my environment if you have any idea.

So far I know if I sudo mkdir under /sys/fs/cgroup/memory, they will generate different folder content between success and fail machine.

Run

cargo test --all test_cgroup --verbose

And I got those error log.

---- sandbox::cgroups::tests::test_cgroup stdout ----
Error: Others("failed to apply cgroup: could not open cgroup file /sys/fs/cgroup/relative/nested/containerd-wasm-shim-test_cgroup/memory.max: No such file or directory (os error 2)")

---- sandbox::cgroups::tests::test_cgroup stdout ----
running test with sudo: sandbox::cgroups::tests::test_cgroup
Error: Stdio(Kind(Other))

After buildingand running the demo example, I got the following error:

sudo ctr run --rm --runtime=io.containerd.wasmtime.v1 docker.io/library/wasmtest:latest testwasm
ctr: Others("Device or resource busy (os error 16)"): unknown

investigation

The task is marked as CREATED:

sudo ctr task ls
TASK          PID    STATUS
testwasm13    0      CREATED

but get the following error when trying to delete it:

sudo ctr task rm testwasm13
ERRO[0000] unable to delete testwasm13                   error="task must be stopped before deletion: created: failed precondition"
ctr: task must be stopped before deletion: created: failed precondition

Also get a slightly different error when trying to "stop" it:

sudo ctr task kill -s SIGKILL testwasm13
ctr: cannot kill non-running container, current state: Exited(TaskState { s: PhantomData }): failed precondition

other info

There seems to be two issues:

• the shim is and containerd are out of sync on the state of the shim. this leads to not being able to clean up the task/container
• There is an unhandled exception in https://github.com/containerd/runwasi/blob/e266bbbb51f2a2e1e207304f87090717278bab45/src/instance.rs#L190 This causes the Device or resource busy (os error 16)

versions

containerd version: containerd containerd.io 1.6.7 0197261a30bf81f1ee8e6a4dd2dea0ef95d67ccb shim version: built from main (e266bbbb51f2a2e1e207304f87090717278bab45) linux version:

lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.1 LTS
Release:        22.04
Codename:       jammy

It does works on my WSL instance:

containerd containerd.io 1.6.6 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1    

 lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.4 LTS
Release:        20.04
Codename:       focal

Cargo seems to really enjoy messing with the tty. Instead of having 2 cargos (main one that executed the test originally, then the sub-test runner) access to the same terminal, make the subprocess go through a pipe for stdout and stderr.

After moving the repo from deislabs to containerd, one of the cgroup tests is failing. See: https://github.com/containerd/runwasi/actions/runs/3597159756/jobs/6058681808.

The repo has a few binaries and a wasi implementation that is fairly tied to wasmtime. #15 makes the core library runtime agnostic, meaning it does not depend on wasmtime.

In order to completely remove wasmtime as a dependency from the core library it may be useful to move the binaries along with the Wasi instance implementation into a separate crate (of course both crates can be in this repo).

This allows tests to fully cleanup. Drop is only implemented for this wrapper because we usually don't want to delete cgroups just because the value went out of scope.

The test wrapper also deletes all parents to make sure the full tree is deleted.

Signed-off-by: James Sturtevant [email protected]

When running make load I ended up not being able to build due to using the docker target in buildx:

error: Docker exporter feature is currently not supported for docker driver. Please switch to a different driver (eg. "docker buildx create --use")
make: *** [Makefile:25: test/out/img.tar] Error 1

This fixes it by using a custom builder and adds a clean target

How do I pass arguments to wasmtime through the shim? I want to use --tcplisten to listen for TCP connections.

I'm trying this command.

ubuntu@wasi:~$ sudo ctr run --rm --runtime=io.containerd.wasmtime.v1 docker.io/martinlinkhorst/wasi:latest wasi10
info: Microsoft.Hosting.Lifetime
      Now listening on: http://localhost:5000
Fatal: TCP accept failed with errno 8. This may mean the host isn't listening for connections. Be sure to pass the --tcplisten parameter.

Modules are compiled and stored in a content addressable store. Two runs of the same module are significantly faster since we only need to digest the wasm module being loaded and do not need to compile it again.

This is still a work in progress. The store is not really concurrency safe yet (it mostly works, but multiple threads/processes could hit issues).

This manually assembles the tar instead of requring buildkit/buildx/docker to create it.

Instead of using buildx to stick our single binary into a container image we can build the tar ourself. This should work on Windows (not tested...) or Linux and only depends on cargo to build.

Closes #36

Containerd Log

time="2022-12-14T09:30:34.243727563Z" level=info msg="CreateContainer within sandbox \"f450952a49060dbe6756fb3638705b7c66404b38b0877741ac319e4edcb825f9\" for container &ContainerMetadata{Name:traefik,Attempt:0,}"
time="2022-12-14T09:30:34.280628605Z" level=info msg="CreateContainer within sandbox \"f450952a49060dbe6756fb3638705b7c66404b38b0877741ac319e4edcb825f9\" for &ContainerMetadata{Name:traefik,Attempt:0,} returns container id \"2a3087458a40f98ef65bbe454da5d84a379f03c1a1e1b19b9b57fd1e3e9885dc\""
time="2022-12-14T09:30:34.281158713Z" level=info msg="StartContainer for \"2a3087458a40f98ef65bbe454da5d84a379f03c1a1e1b19b9b57fd1e3e9885dc\""
time="2022-12-14T09:30:34.350862636Z" level=info msg="StartContainer for \"2a3087458a40f98ef65bbe454da5d84a379f03c1a1e1b19b9b57fd1e3e9885dc\" returns successfully"
time="2022-12-14T09:30:41.365630407Z" level=info msg="CreateContainer within sandbox \"cb2719f323623808ff663e1d0e409530a160cb62e702d0a1c3bc8670046e57fd\" for container &ContainerMetadata{Name:testwasm,Attempt:2,}"
time="2022-12-14T09:30:41.417023160Z" level=info msg="CreateContainer within sandbox \"cb2719f323623808ff663e1d0e409530a160cb62e702d0a1c3bc8670046e57fd\" for &ContainerMetadata{Name:testwasm,Attempt:2,} returns container id \"6ebb8cc29b333a124661983fba2dec5c82e4fc32c9a484212de41e4a3fa1e06e\""
time="2022-12-14T09:30:41.417626869Z" level=info msg="StartContainer for \"6ebb8cc29b333a124661983fba2dec5c82e4fc32c9a484212de41e4a3fa1e06e\""
[INFO] starting instance
[INFO] preparing module
[INFO] opening rootfs
[INFO] setting up wasi
[INFO] opening stdin
[INFO] opening stdout
[INFO] opening stderr
[INFO] building wasi context
[INFO] wasi context ready
[INFO] loading module from file
[INFO] instantiating instnace
[INFO] getting start function
[INFO] starting wasi instance
[INFO] started wasi instance with tid 1794
time="2022-12-14T09:30:41.559211243Z" level=info msg="StartContainer for \"6ebb8cc29b333a124661983fba2dec5c82e4fc32c9a484212de41e4a3fa1e06e\" returns successfully"
[INFO] child 1794 killed by signal SIGKILL, dumped: false
[INFO] wasi instance exited with status 137
time="2022-12-14T09:30:43.108591141Z" level=info msg="shim disconnected" id=6ebb8cc29b333a124661983fba2dec5c82e4fc32c9a484212de41e4a3fa1e06e
time="2022-12-14T09:30:43.108722243Z" level=warning msg="cleaning up after shim disconnected" id=6ebb8cc29b333a124661983fba2dec5c82e4fc32c9a484212de41e4a3fa1e06e namespace=k8s.io
time="2022-12-14T09:30:43.108732343Z" level=info msg="cleaning up dead shim"
time="2022-12-14T09:30:44.500146327Z" level=info msg="RemoveContainer for \"82de028e9dba19dfe45615e0efaa1e73cf35d05734b09aade8489485c5f48a84\""
time="2022-12-14T09:30:44.517400480Z" level=info msg="RemoveContainer for \"82de028e9dba19dfe45615e0efaa1e73cf35d05734b09aade8489485c5f48a84\" returns successfully"
time="2022-12-14T09:31:12.364643823Z" level=info msg="CreateContainer within sandbox \"cb2719f323623808ff663e1d0e409530a160cb62e702d0a1c3bc8670046e57fd\" for container &ContainerMetadata{Name:testwasm,Attempt:3,}"
time="2022-12-14T09:31:12.398472900Z" level=info msg="CreateContainer within sandbox \"cb2719f323623808ff663e1d0e409530a160cb62e702d0a1c3bc8670046e57fd\" for &ContainerMetadata{Name:testwasm,Attempt:3,} returns container id \"0101352d7327f58fc458166c0df7ce439528db33bd5006da002e69bb33d218d0\""
time="2022-12-14T09:31:12.398916606Z" level=info msg="StartContainer for \"0101352d7327f58fc458166c0df7ce439528db33bd5006da002e69bb33d218d0\""
[INFO] starting instance
[INFO] preparing module
[INFO] opening rootfs
[INFO] setting up wasi
[INFO] opening stdin
[INFO] opening stdout
[INFO] opening stderr
[INFO] building wasi context
[INFO] wasi context ready
[INFO] loading module from file
[INFO] instantiating instnace
[INFO] getting start function
[INFO] starting wasi instance
[INFO] started wasi instance with tid 1862
time="2022-12-14T09:31:12.528460632Z" level=info msg="StartContainer for \"0101352d7327f58fc458166c0df7ce439528db33bd5006da002e69bb33d218d0\" returns successfully"
[ERROR] error waiting for pid 1862: ECHILD: No child processes

Notice that there is a log message says "[INFO] child 1794 killed by signal SIGKILL, dumped: false"

How to reproduce?

Setup a k3d cluster image follow the steps in https://github.com/deislabs/containerd-wasm-shims/tree/main/deployments/k3d. Replace the spin & slight shim with wasmtime shim in "config.toml.tmpl"

[plugins.cri.containerd.runtimes.wasmtime]
  runtime_type = "io.containerd.wasmtime.v1"

Once the k3d cluster image is created, we can create a k3d cluster by running k3d cluster create k3s-default --image k3swithshim --api-port 6550 -p "8081:80@loadbalancer" --agents 1

Then apply the following workloads

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: wasmtime
handler: wasmtime

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wasm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: wasm
  template:
    metadata:
      labels:
        app: wasm
    spec:
      runtimeClassName: wasmtime
      containers:
        - name: testwasm
          image: docker.io/mossaka/wasmtest:2

I don't think the project has a logo, so I'm proposing the following.

I'm excited about the project, but I'm not a developer, so this is me trying to contribute. The idea is running WASI :-D

I won't lose any sleep if nobody likes it or is too busy to care right now.