A Rancher and Kubernetes optimized immutable Linux distribution based on openSUSE

RancherOS v2

WORK IN PROGRESS

RancherOS v2 is an immutable Linux distribution built to run Rancher and it's corresponding Kubernetes distributions RKE2 and k3s. It is built using the cOS-toolkit and based on openSUSE. Initial node configurations is done using only a cloud-init style approach and all further maintenance is done using Kubernetes operators.

Use Cases

RancherOS is intended to be ran as the operating system beneath a Rancher Multi-Cluster Management server or as a node in a Kubernetes cluster managed by Rancher. RancherOS also allows you to build stand alone Kubernetes clusters that run an embedded and smaller version of Rancher to manage the local cluster. A key attribute of RancherOS is that it is managed by Rancher and thus Rancher will exist either locally in the cluster or centrally with Rancher Multi-Cluster Manager.

Architecture

OCI Image based

RancherOS v2 is an A/B style image based distribution. One first runs on a read-only image A and to do an upgrade pulls a new read only image B and then reboots the system to run on B. What is unique about RancherOS v2 is that the runtime images come from OCI Images. Not an OCI Image containing special artifacts, but an actual Docker runnable image that is built using standard Docker build processes. RancherOS is built using normal docker build and if you wish to customize the OS image all you need to do is create a new Dockerfile.

rancherd

RancherOS v2 includes no container runtime, Kubernetes distribution, or Rancher itself. All of these assests are dynamically pulled at runtime. All that is included in RancherOS is rancherd which is responsible for bootstrapping RKE2/k3s and Rancher from an OCI registry. This means an update to containerd, k3s, RKE2, or Rancher does not require an OS upgrade or node reboot.

cloud-init

RancherOS v2 is initially configured using a simple version of cloud-init. It is not expected that one will need to do a lot of customization to RancherOS as the core OS's sole purpose is to run Rancher and Kubernetes and not serve as a generic Linux distribution.

RancherOS Operator

RancherOS v2 includes an operator that is responsible for managing OS upgrades and assiting with secure device onboarding (SDO).

openSUSE Leap

RancherOS v2 is based off of openSUSE Leap. There is no specific tie in to openSUSE beyond that RancherOS assumes the underlying distribution is based on systemd. We choose openSUSE for obvious reasons, but beyond that openSUSE Leap provides a stable layer to build upon that is well tested and has paths to commercial support, if one chooses.

Comments
  • Server is not automatically reboot after installation

    Server is not automatically reboot after installation

    I generated a RancherOS iso with elemental-image-build from https://github.com/rancher-sandbox/os2-opensuse-image, then installed a VM with it. Installation went fine but there was no automatic reboot at the end, and this should be the default behaviour.

  • Is there any Roadmap for RancherOS v2?

    Is there any Roadmap for RancherOS v2?

    I'm very interested in RancherOS v2. I have also seen that the alpha release phase has already started :partying_face: . Is there any roadmap / timeframe for this release? When will any beta / rc / stable version be available? When could/should RancherOS v2 be used instead of RancherOS v1? Will there be any upgrade path from RancherOS v1 to RancherOS v2?

    Thank you

    ping: rancher/os#3063

  • creating users with cloud init not functional

    creating users with cloud init not functional

    Hello!

    I'm unable to log in with the user created, I've tried both in isolation as well with no avail. I'm using the ros-image-build with an iso output and running on a physical machine. Seems potentially related to https://github.com/rancher/os2/issues/11 https://github.com/rancher/os2/issues/10

    Given this cloud-config:

    #cloud-config
    hostname: "ros-001"
    users:
    - name: "kampe"
      passwd: "password"
      groups: "users"
      ssh_authorized_keys:
      - github:kampe
    
    ssh_authorized_keys:
      - github:kampe
    
    runcmd:
    - whoami
    
    rancherd:
      kubernetesVersion: v1.22.2+k3s1
      rancherVersion: v2.6.0
      rancherValues:
        features: multi-cluster-management=false
        hostPort: 8443
        ingress:
          enabled: false
        noDefaultAdmin: true
        replicas: -3
        tls: external
     
      token: sometoken
    
      discovery:
        params:
          provider: "mdns"
          service: "rancher-server"
        expectedServers: 3
        serverCacheDuration: 1m
      role: cluster-init
    
      nodeName: ros-001
    
  • Booting from openstack doesn't load userdata

    Booting from openstack doesn't load userdata

    I've tried to apply the following cloud-init in the custom user-data when creating a VM in openstack from the os2 qcow alpha12 release:

    #cloud-config
    rancherd:
      role: cluster-init
      rancherValues:
        features:
        - multi-cluster-management=true
    ssh_authorized_keys:
      - github:mudler
    runcmd:
      - curl -fL https://raw.githubusercontent.com/rancher/rancherd/master/install.sh | sh -
    

    This seems to work locally, but when the machine starts from openstack I cannot login, seems ignoring my keys

  • TPM device is a hard requirement

    TPM device is a hard requirement

    Can't join nodes that don't have a TPM device, getting the following error when calling ros-installer -config-file /oem/userdata.yaml:

    ERRO[0000] failed to read registration URL https://xx.lan/v1-rancheros/registration/xxxx, retrying: opening tpm: TPM device not available
    

    This is an issue at least for running vagrant images as VirtualBox currently doesn't support TPM, but applies to other hypervisors including raspberrypi4 and baremetal which don't have the TPM hardware in general

    Note, this doesn't seem to be an issue for libvirt/qemu: https://documentation.suse.com/sles/15-SP3/html/SLES-all/tpm.html https://github.com/stefanberger/swtpm/issues/33

    See also: https://github.com/stefanberger/swtpm/issues/33

  • QR Code for bootstrapping

    QR Code for bootstrapping

    It would be really nice to be able to be able to scan a QR code that provides Rancher with all the onboarding data from a newly bootstrapped machine.

    The workflow would look like:

    • Boot installer on target machine
    • After laying down partitions and rebooting, display QR code with any data needed
    • User then logs into Rancher on their phone or tablet
    • User browses to bootstrapping tab and clicks on "add machine with QR code"
    • Rancher opens camera
    • User scans code
    • Rancher stores data and now can control device securely
  • Feature Request: Allow Operator to manage being a downstream

    Feature Request: Allow Operator to manage being a downstream "cattle"

    Hello!

    Problem: There's seemingly no great way to manage the registration of a "edge" rancher cluster with a cloud hosted "multicluster-managment" rancher - or if there is I'm missing something fundamental about how I should be managing these "cattle" clusters I create downstream of the Rancher.

    Current Solution: We manage this ourselves by creating a k8s job that runs at runtime and utilizes a "service account" with credentials retrieved from centralized secret store to create and register the cluster with rancher "multicluster-management" which then self applies the cattle-agent yaml.

    is there a more preferred way to manage these types of registrations with the rancher?

  • Automatic installation not picked up in vsphere

    Automatic installation not picked up in vsphere

    I don't have logs to attach as I didn't had access to the machine, but seems that userdata from ISO didn't work correctly under vsphere. The automatic installation and ros-installer didn't catch the userdata and thus automatic installation was not kicking while joining nodes.

    The /oem/userdata files were populated correctly, workaround was to point the ros-installer to the userdata file directly with -config-file

    This card is mostly to try to reproduce and track the issue

  • Support container images as installation bootstrap

    Support container images as installation bootstrap

    cOS-toolkit gained the capability to bootstrap directly container images besides isos, so it should be possible to enhance the current mechanism that applies to ISOs to container images as well.

    See for a reference isoUrl

  • Broken cloud-init after install

    Broken cloud-init after install

    I'm trying to install os2 to bring a standalone K8s cluster to register other nodes to it, by using the following cloud-config file:

    #cloud-config
    rancherd:
      role: cluster-init
      rancherValues:
        features:
        - multi-cluster-management=true
    ssh_authorized_keys:
      - github:mudler
    runcmd:
      - curl -fL https://raw.githubusercontent.com/rancher/rancherd/master/install.sh | sh -
    

    And by running after booting the ISO

    $> ros-installer -config-file cloud-init.yaml
    

    However, after installation, the generated cloud-init config in /oem/99_custom.yaml is the following: VirtualBox_test_13_01_2022_11_15_46

    which breaks the cloud-init parsing reference, hence can't login after boot, see https://rancher.github.io/os2/configuration/

  • Slack channel?

    Slack channel?

    Hi, interesting project and I've started a matchbox/terraform test of OS2. @ibuildthecloud ,will you consider adding a slack channel? (Sorry for a question issue)

    /hw

Related tags
Truly Minimal Linux Distribution for Containers

Statesman Statesman is a minimal Linux distribution, running from memory, that has just enough functionality to run OCI-compatible containers. Rationa

Nov 12, 2021
A Kubernetes operator that allows for automatic provisioning and distribution of cert-manager certs across namespaces

cached-certificate-operator CachedCertificate Workflow When a CachedCertificate is created or updated the operator does the following: Check for a val

Sep 6, 2022
Assigns floating ip addresses to Rancher Guest clusters.
Assigns floating ip addresses to Rancher Guest clusters.

kube-fip-operator The kube-fip-operator application manages the FloatingIP and FloatingIPRange Custom Resource Definition objects in a Rancher environ

Dec 6, 2021
repo de teste para executar á pipeline do rancher

pipeline-example-go This is a sample golang project to demonstrate the integration with rancher pipeline. Building go build -o ./bin/hello-server Runn

Dec 19, 2021
Fast docker image distribution plugin for containerd, based on CRFS/stargz
Fast docker image distribution plugin for containerd, based on CRFS/stargz

[ ⬇️ Download] [ ?? Browse images] [ ☸ Quick Start (Kubernetes)] [ ?? Quick Start (nerdctl)] Stargz Snapshotter Read also introductory blog: Startup C

Dec 29, 2022
immutable, fluent, builders for Kubernetes resources

Dies - immutable, fluent, builders for Kubernetes resources Using dies Common methods Creating dies diegen die markers +die This project contains dies

May 6, 2022
Open Source runtime scanner for Linux containers (LXD), It performs security audit checks based on CIS Linux containers Benchmark specification
Open Source runtime scanner for Linux containers (LXD), It performs security audit checks based on CIS Linux containers  Benchmark specification

lxd-probe Scan your Linux container runtime !! Lxd-Probe is an open source audit scanner who perform audit check on a linux container manager and outp

Dec 26, 2022
a small form factor OpenShift/Kubernetes optimized for edge computing

Microshift Microshift is OpenShift1 Kubernetes in a small form factor and optimized for edge computing. Edge devices deployed out in the field pose ve

Dec 29, 2022
resource manifest distribution among multiple clusters.

Providing content to managed clusters Support a primitive that enables resources to be applied to a managed cluster. Community, discussion, contributi

Dec 26, 2022
Walker's alias method is an efficient algorithm to sample from a discrete probability distribution.

walker-alias Walker's alias method is an efficient algorithm to sample from a discrete probability distribution. This means given an arbitrary probabi

Jun 14, 2022
Apachedist-resource - A concourse resource to track updates of an apache distribution, e.g. tomcat

Apache Distribution Resource A concourse resource that can track information abo

Feb 2, 2022
Tape backup software optimized for large WORM data and long-term recoverability

Mixtape Backup software for tape users with lots of WORM data. Draft design License This codebase is not open-source software (or free, or "libre") at

Oct 30, 2022
Go library providing algorithms optimized to leverage the characteristics of modern CPUs

asm Go library providing algorithms optimized to leverage the characteristics of modern CPUs. Motivation With the development of Cloud technologies, a

Dec 29, 2022
Kubernetes OS Server - Kubernetes Extension API server exposing OS configuration like sysctl via Kubernetes API

KOSS is a Extension API Server which exposes OS properties and functionality using Kubernetes API, so it can be accessed using e.g. kubectl. At the moment this is highly experimental and only managing sysctl is supported. To make things actually usable, you must run KOSS binary as root on the machine you will be managing.

May 19, 2021
Linux Controllers for Kubernetes

Tambourine Kubelet replacement with Built in Linux extensions Development Success: Install, Manage, and Observe a new systemd service from Kubernetes.

Jun 2, 2021
KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes
 KEDA is a Kubernetes-based Event Driven Autoscaling component. It provides event driven scale for any container running in Kubernetes

Kubernetes-based Event Driven Autoscaling KEDA allows for fine-grained autoscaling (including to/from zero) for event driven Kubernetes workloads. KED

Jan 7, 2023
An Easy to use Go framework for Kubernetes based on kubernetes/client-go

k8devel An Easy to use Go framework for Kubernetes based on kubernetes/client-go, see examples dir for a quick start. How to test it ? Download the mo

Mar 25, 2022
Litmus helps Kubernetes SREs and developers practice chaos engineering in a Kubernetes native way.
Litmus helps Kubernetes SREs and developers practice chaos engineering in a Kubernetes native way.

Litmus Cloud-Native Chaos Engineering Read this in other languages. ???? ???? ???? ???? Overview Litmus is a toolset to do cloud-native chaos engineer

Jan 1, 2023
vcluster - Create fully functional virtual Kubernetes clusters - Each cluster runs inside a Kubernetes namespace and can be started within seconds
vcluster - Create fully functional virtual Kubernetes clusters - Each cluster runs inside a Kubernetes namespace and can be started within seconds

Website • Quickstart • Documentation • Blog • Twitter • Slack vcluster - Virtual Clusters For Kubernetes Lightweight & Low-Overhead - Based on k3s, bu

Jan 4, 2023