The goal of Lima is to promote containerd including nerdctl (contaiNERD ctl) to Mac users, but Lima can be used for non-container applications as well.
Adopters
Container environments:
Rancher Desktop: Kubernetes and container management to the desktop
Colima: Docker (and Kubernetes) on macOS with minimal setup
GUI:
Lima xbar plugin: xbar plugin to start/stop VMs from the menu bar and see their running status.
$ uname -aDarwin macbook.local 20.4.0 Darwin Kernel Version 20.4.0: Thu Apr 22 21:46:47 PDT 2021; root:xnu-7195.101.2~1/RELEASE_X86_64 x86_64
$ lima uname -aLinux lima-default 5.11.0-16-generic #17-Ubuntu SMP Wed Apr 14 20:12:43 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
$ LIMA_INSTANCE=arm lima uname -aLinux lima-arm 5.11.0-16-generic #17-Ubuntu SMP Wed Apr 14 20:10:16 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux
Sharing files across macOS and Linux
some-file $ lima cat some-file files under /Users on macOS filesystem are readable from Linux $ lima sh -c 'echo "/tmp/lima is writable from both macOS and Linux" > /tmp/lima/another-file' $ cat /tmp/lima/another-file /tmp/lima is writable from both macOS and Linux ">
$ echo"files under /Users on macOS filesystem are readable from Linux"> some-file
$ lima cat some-filefiles under /Users on macOS filesystem are readable from Linux
$ lima sh -c 'echo "/tmp/lima is writable from both macOS and Linux" > /tmp/lima/another-file'
$ cat /tmp/lima/another-file/tmp/lima is writable from both macOS and Linux
Running containerd containers (compatible with Docker containers)
$ lima nerdctl run -d --name nginx -p 127.0.0.1:8080:80 nginx:alpine
To install Lima from the source, run make && make install.
NOTE Lima is not regularly tested on ARM Mac (due to lack of CI).
Usage
[macOS]$ limactl start...INFO[0029] READY. Run `lima` to open the shell.[macOS]$ lima unameLinux
Detailed usage:
Run limactl start
to start the Linux instance. The default instance name is "default". Lima automatically opens an editor (vi) for reviewing and modifying the configuration. Wait until "READY" to be printed on the host terminal.
Run limactl shell
to launch on Linux. For the "default" instance, this command can be shortened as lima . The lima command also accepts the instance name as the environment variable $LIMA_INSTANCE.
Run limactl copy ...
to copy files between instances, or between instances and the host. Use
:
to specify a source or target inside an instance.
Run limactl list [--json] to show the instances.
Run limactl stop [--force]
to stop the instance.
Run limactl delete [--force]
to delete the instance.
To enable bash completion, add source <(limactl completion bash) to ~/.bash_profile.
To enable zsh completion, see limactl completion zsh --help
⚠️
CAUTION: make sure to back up your data
Lima may have bugs that result in loss of data.
Make sure to back up your data before running Lima.
Especially, the following data might be easily lost:
Data in the shared writable directories (/tmp/lima by default), probably after hibernation of the host machine (e.g., after closing and reopening the laptop lid)
Data in the VM image, mostly when upgrading the version of lima
The default Ubuntu image also contains LXD. Runlima sudo lxc init to set up LXD.
See also third party containerd projects based on Lima:
Rancher Desktop: Kubernetes and container management to the desktop
Colima: Docker (and Kubernetes) on macOS with minimal setup
"Can I run Lima with a remote Linux machine?"
Lima itself does not support connecting to a remote Linux machine, but sshocker, the predecessor or Lima, provides similar features for remote Linux machines.
e.g., run sshocker -v /Users/foo:/home/foo/mnt -p 8080:80
@
to expose /Users/foo to the remote machine as /home/foo/mnt, and forward localhost:8080 to the port 80 of the remote machine.
"Advantages compared to Docker for Mac?"
Lima is free software (Apache License 2.0), while Docker for Mac is not. Their EULA even prohibits disclosure of benchmarking result.
On the other hand, Moby, aka Docker for Linux, is free software, but Moby/Docker lacks several novel features of containerd, such as:
If you have installed QEMU v6.0.0 or later on macOS 11 via homebrew, your QEMU binary should have been already automatically signed to enable HVF acceleration.
However, if you see HV_ERROR, you might need to sign the binary manually.
if you are on macOS 10.15.7 or 11.0 or later make sure the entitlement com.apple.vm.hypervisor is not added. It only works on older macOS versions. You can clear the codesigning with codesign --remove-signature /usr/local/bin/qemu-system-x86_64 and start over.
"QEMU crashes with vmx_write_mem: mmu_gva_to_gpa XXXXXXXXXXXXXXXX failed"
This error is known to happen when running an image of RHEL8-compatible distribution such as CentOS 8 on Intel Mac. A workaround is to set environment variable QEMU_SYSTEM_X86_64="qemu-system-x86_64 -cpu Haswell-v4".
Prior to Lima v0.7.0, Lima did not support forwarding privileged ports (1-1023). e.g., you had to use 8080, not 80.
Lima v0.7.0 and later supports forwarding privileged ports on macOS hosts.
On Linux hosts, you might have to set sysctl value net.ipv4.ip_unprivileged_port_start=0.
stuck on "Waiting for the essential requirement 1 of X: "ssh"
libslirp v4.6.0 used by QEMU is known to be broken. If you have libslirp v4.6.0 in /usr/local/Cellar/libslirp, you have to upgrade it to v4.6.1 or later (brew upgrade).
"permission denied" for limactl cp command
The copy command only works for instances that have been created by lima 0.5.0 or later. You can manually install the required identity on older instances with (replace INSTANCE with actual instance name):
< ~/.lima/_config/user.pub limactl shell INSTANCE sh -c 'tee -a ~/.ssh/authorized_keys'
Networking
"Cannot access the guest IP 192.168.5.15 from the host"
The default guest IP 192.168.5.15 is not accessible from the host and other guests.
To add another IP address that is accessible from the host and other virtual machines, enable vde_vmnet.
~~QEMU with 9p-darwin patches available here:~~
~~https://github.com/afbjorklund/homebrew-core/blob/qemu-9p-darwin/Formula/qemu.rb~~
~~Patches from: https://github.com/willcohen/qemu/commits/v6.2.0-9p-darwin (or use HEAD)~~
~~You can also use the latest/greatest QEMU HEAD.~~
EDIT: The qemu in brew now supports virtfs, also for darwin systems.
Example mount when booting with "sshfs":
:/tmp/lima on /tmp/lima type fuse.sshfs (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000,allow_other)
Example mount when booting with "9p":
mount1 on /tmp/lima type 9p (rw,sync,dirsync,relatime,access=client,msize=131072,trans=virtio)
After the display is shown, press the Esc key several times to show the GRUB prompt for choosing an older kernel.
You can also mount the disk image from another instance to modify the GRUB config or rescue the data inside the image.
https://github.com/lima-vm/lima/issues/712#issuecomment-1065266828
See also https://gitlab.com/qemu-project/qemu/-/issues/899
:pray: Help wanted for testing
This QEMU patch may fix the issue: https://github.com/AkihiroSuda/homebrew-core/commit/177d583c741f4f50a12f326f632c54dbf169d37b
(from https://gitlab.com/qemu-project/qemu/-/issues/899)
Download qemu.rb and run brew install --build-from-source ./qemu.rb to apply the patch.
Below is the original post by @mateka
Description
Today I was working in lima VM (Ubuntu 20.04; Mac M1). After a while, it has lost connection (as it often does), so I had tried to restart it. Old VM and even brand new will not start and return an error after:
Waiting for the essential requirement 1 of 5: "ssh"
limactl --debug start VM-name prints:
DEBU[0171] [hostagent] executing ssh for script "ssh": /usr/bin/ssh [ssh -F /dev/null -o IdentityFile="/Users/matek/.lima/_config/user" -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o NoHostAuthenticationForLocalhost=yes -o GSSAPIAuthentication=no -o PreferredAuthentications=publickey -o Compression=no -o BatchMode=yes -o IdentitiesOnly=yes -o Ciphers="^[email protected],[email protected]" -o User=matek -o ControlMaster=auto -o ControlPath="/Users/matek/.lima/test/ssh.sock" -o ControlPersist=5m -p 52070 127.0.0.1 -- /bin/bash]
DEBU[0246] [hostagent] stdout="", stderr="kex_exchange_identification: read: Connection reset by peer\r\nConnection reset by 127.0.0.1 port 52070\r\n", err=failed to execute script "ssh": stdout="", stderr="kex_exchange_identification: read: Connection reset by peer\r\nConnection reset by 127.0.0.1 port 52070\r\n": exit status 255
Rancher Desktop seems to work.
Set instance name while starting
Running limactl start foo.yml automatically sets the instance name to foo. It would be nice to be able to set a name different than the file name. Ex. running limactl start foo.yml --name bar would create an instance named bar.
Support for Virtualization.Framework for macOS 13
This PR provides support for using Virtualization.Framework as a optional driver.
The following are the changes done related to using drivers,
[x] Driver interface
[x] Migrate current QEMU implementation to driver
[x] New driver vz for Virtualization.Framework
The following are the features of lima, the vz driver should provide support for below,
[x] Running VM (Using both disk img and iso)*
[x] Slirp network for guest to host communication (uses gvisor-tap-vsock)
[x] Host to guest network (uses vz NAT gateway)*
[x] Directory sharing, supports reverse-sshfs and virtiofs (newly added)
[x] Port forwarding
[x] Host DNS resolver
[ ] Display (Deferring it for now as it requires runtime.LockOsThread() to be called in the beginning of hostagent start cmd)
Notes
Vz driver internally converts qcow to raw image using (qemu-img convert). This is because vz only supports raw disk
Need to provide yaml configuration for NAT
Know Issues
Serial log doesn't contain system boot logs
Gvisor-tap-vsock will throw some errors like [e.connection](tcpproxy:) during start-up (This is because of port 22 Forwards being tried before VM is ready) [Not a failure/blocker just info message thrown, but we can look into fixing in a follow-up to call ssh forward manually]
When vz vm stops, it doesn't trigger hostagent stop [Fixed]
Testing
Tested the following templates with driver: vz. All test are done on macOS 13 intel as of now.
[x] almalinux.yaml
[x] alpine.yaml
[x] apptainer.yaml
[ ] archlinux.yaml
[x] buildkit.yaml
[ ] centos-stream.yaml
[x] debian.yaml
[x] docker.yaml
[x] faasd.yaml
[x] fedora.yaml
[x] k3s.yaml
[x] k8s.yaml
[x] nomad.yaml
[ ] opensuse.yaml
[ ] oraclelinux.yaml
[x] podman.yaml
[ ] rocky.yaml
[x] ubuntu.yaml
[x] vmnet.yaml
`host.lima.internal` should be defined *inside* running containers
A note though - host.lima.internal should be defined inside running containers. That's what docker desktop has always done, it provides that name resolution. People don't really want to get to the host just from inside the lima container, they want to get to the host from inside containers they're running.
Originally posted by @rfay in https://github.com/lima-vm/lima/discussions/389#discussioncomment-1855625
Replace reverse SSHFS with Samba
🔴 Current blocker: nls_utf8.ko is missing in openSUSE 15.3 https://bugzilla.opensuse.org/show_bug.cgi?id=1190797
🔴 I also have to rewrite this PR to make Samba non-default (https://github.com/lima-vm/lima/pull/188#discussion_r718677030)
See the changes of docs/internal.md for the design.
On macOS hosts, /usr/local/sbin/samba-dot-org-smbd is used as the smbd binary.
This binary can be installed with brew install samba.
Apple's version of /usr/sbin/smbd cannot be used.
The binary path can be overridden with $SMBD env var.
smbd is connected to QEMU via smb's stdio.
The samba address 192.168.5.4:445 is only accessible from the guest, not from the host.
When the host's hostname is not present in /etc/hosts on the host filesystem, the Lima hostagent launches a mDNS to help looking up the hostname.
Otherwise starting Samba takes 25 secs with "getaddrinfo failed" error.
See the pkg/samba/getaddrinfoworkaround package for the further information.
How to install containerd with alpine-lima
It's not fully clear how to add containerd to alpine, since the scripts require systemd:
ERRO[0018] [1 error occurred:
* failed to satisfy the optional requirement 1 of 2 "systemd must be available": systemd is required to run containerd, but does not seem to be available.
Make sure that you use an image that supports systemd. If you do not want to run
containerd, please make sure that both 'container.system' and 'containerd.user'
are set to 'false' in the config file.
Installing containerd is quite simple, but packages for "nerdctl" and "buildkit" are missing...
socket_vmnet is similar to vde_vmnet but does not depend on VDE.
https://github.com/lima-vm/socket_vmnet
See docs/network.md for how to create networks.yaml with socketVMNet.
When both socketVMNet and vdeVMNet (deprecated) are present in the YAML, socketVMNet is chosen.
Help wanted for testing https://github.com/lima-vm/lima/pull/703 on M1 macOS
Install Lima v0.9.0-beta.0 or later (brew install --HEAD lima, or git clone https://github.com/lima-vm/lima.git && cd lima && make && make install)
Install QEMU v6.2.0 (brew install qemu)
Create instances with the following memory: 8 GiB, 4 GiB, 2 GiB.
$ limactl start
? Creating an instance "default" [Use arrows to move, type to filter]
Proceed with the current configuration
> Open an editor to review or modify the current configuration
Choose another example (docker, podman, archlinux, fedora, ...)
Exit
(Change "memory: null" to "memory: 8 GiB" in the editor, and confirm that the instance works)
$ limactl delete -f defaut
Install QEMU master (brew install --HEAD qemu)
Repeat the step 3
Enable trimming of the `diffdisk`
The diffdisk starts out small, but grows quickly as users create and delete container images. This can become an issue on laptops with limited free space.
I tried to enable trim support just for Alpine, for testing, but couldn't get it to work. Here is the patch I tried:
--- pkg/cidata/cidata.TEMPLATE.d/boot/05-persistent-data-volume.sh
+++ pkg/cidata/cidata.TEMPLATE.d/boot/05-persistent-data-volume.sh
@@ -14,7 +14,7 @@ DATADIRS="/etc /home /tmp /usr/local /var/lib"
if [ "$(awk '$2 == "/" {print $3}' /proc/mounts)" == "tmpfs" ]; then
mkdir -p /mnt/data
if [ -e /dev/disk/by-label/data-volume ]; then
- mount -t ext4 /dev/disk/by-label/data-volume /mnt/data
+ mount -t ext4 -o discard /dev/disk/by-label/data-volume /mnt/data
else
# Find an unpartitioned disk and create data-volume
DISKS=$(lsblk --list --noheadings --output name,type | awk '$2 == "disk" {print $1}')
@@ -32,7 +32,7 @@ if [ "$(awk '$2 == "/" {print $3}' /proc/mounts)" == "tmpfs" ]; then
echo 'type=83' | sfdisk --label dos /dev/"${DISK}"
PART=$(lsblk --list /dev/"${DISK}" --noheadings --output name,type | awk '$2 == "part" {prin
mkfs.ext4 -L data-volume /dev/"${PART}"
- mount -t ext4 /dev/disk/by-label/data-volume /mnt/data
+ mount -t ext4 -o discard /dev/disk/by-label/data-volume /mnt/data
for DIR in ${DATADIRS}; do
DEST="/mnt/data$(dirname "${DIR}")"
mkdir -p "${DIR}" "${DEST}"
--- pkg/qemu/qemu.go
+++ pkg/qemu/qemu.go
@@ -261,7 +261,7 @@ func Cmdline(cfg Config) (string, []string, error) {
args = appendArgsIfNoConflict(args, "-boot", "order=c,splash-time=0,menu=on")
}
if diskSize, _ := units.RAMInBytes(cfg.LimaYAML.Disk); diskSize > 0 {
- args = append(args, "-drive", fmt.Sprintf("file=%s,if=virtio", diffDisk))
+ args = append(args, "-drive", fmt.Sprintf("file=%s,if=virtio,discard=unmap", diffDisk))
} else if !isBaseDiskCDROM {
args = append(args, "-drive", fmt.Sprintf("file=%s,if=virtio", baseDisk))
}
I've created some files via dd if=/dev/urandom of=1.bin bs=64M count=64 iflag=fullblock etc and verified the growth in diffdisk size.
I wonder if this is a macOS limitation, that qemu doesn't implement the sparse file logic for APFS.
Thoughts?
VirtualBox driver, for legacy compatibility and portability
Description
Once upon a time, there was something called Docker Machine and it was running on VirtualBox.
It did have some other drivers, like qemu, but the virtualbox driver was the portable and default...
Now that Lima has a driver framework, it could be useful to have a "vbox" driver next to the "qemu" ?
It does support features like host networking and shared folders, that QEMU is still struggling with.
Pros: might run on more platforms, such as legacy releases or other operating systems
Cons: only supports Intel images, no support for COW images (resulting in larger files)
But the biggest advantage is having another driver that also works on non-Mac hosts.
And it can help to improve the driver API, if testing with more different implementations
Lima get stuck when certain frequency of DNS resolution occurs
Description
Actual Behavior
When I continue to execute nslookup/ping/dig or any other DNS name resolution operations within Lima, Lima get stuck after a certain number of times (lasting about 4 minutes).
This issue also occurs inside Docker container when using Rancher Desktop, so this is critial issue for any system that uses container linking, like Apache+PHP and MySQL.
Steps to Reproduce
Login to Lima and keep running nslookup.
$ rdctl shell
lima-rancher-desktop:/Users/xxx$ while true; do nslookup www.google.co.jp; done
While the above process is running, show a list of UDP open files that qemu-system-aarch64 handles on host OS.
Example of the actual issue. If Ctrl-C is sent to the terminal, where limactl was launched, which in order started limactl host agent, it will kill the host agent. Or if you close the console window it will also terminate the HA, which is not expected behavior.
Related to https://github.com/lima-vm/lima/issues/909
`limactl inspect --format json` and `limactl ls --format json` produces the exactly same output
Noticed that limactl inspect --format json and limactl ls --format json produces the exactly same output now.
Probably, limactl ls isn't expected to print the config field?
Originally posted by @AkihiroSuda in https://github.com/lima-vm/lima/issues/1249#issuecomment-1372978716
Support for VBox driver
This is a legacy driver for Oracle VirtualBox, x86_64 only.
It also has support for shared folders and hostonly networks,
but they are not implemented just yet (sshfs and nat only)
Deletion does not work properly, due to the VM "registry".
visualize qemu vm statistics in real-time
Description
We (w/@dentrax) thought it'd be good to create a sub-command like stats which would display the VM statistics in real-time as the docker's stats command.
# can take a VM name as an argument or will list all the VM stats if no VM name is given
$ limactl stats <VM>
NAME CPUS MEMORY DISK
vbox %13 %27 %31
![[Ubuntu on M1] Lima VM will not start (`EFI stub: ERROR: FIRMWARE BUG: kernel image not aligned on 64k boundary`)](https://avatars.githubusercontent.com/u/6030795?v=4)
