Sabakan is a versatile network boot server designed for large on-premise data centers.

GitHub release main Go Reference Go Report Card

Sabakan

sabakan architecture

Sabakan is a versatile network boot server designed for large on-premise data centers. Currently, it is made only for Flatcar Container Linux.

Project Status: GA (General Availability)

Features

  • High availability

    High availability of sabakan is just as easy as running multiple sabakan servers.

    Sabakan data are stored and shared in etcd. For example, DHCP lease information are shared between sabakan instances to avoid conflicts.

  • Machine inventory with IPAM (IP address management)

    Sabakan keeps an inventory of machines in a data center. Their IP addresses are automatically assigned by sabakan.

  • DHCP service

    Sabakan provides DHCP service that supports UEFI HTTP Boot and iPXE HTTP Boot. It also supports DHCP relay request to make DHCP service highly available.

  • HTTP service (network file server)

    Sabakan provides HTTP service for network boot clients. Users can upload any kind of files other than OS images to sabakan. Clients can download them to initialize the system after boot.

  • Template system for Ignition

    Ignition is a boot provisioning system for Flatcar Container Linux. Ignition configuration is not friendly for operators as it is written in a plain JSON.

    Sabakan provides a friendly and super versatile template system for Ignition configurations. For each client machine, sabakan renders Ignition configuration from templates.

  • Life-cycle management

    Machines in the inventory has a life-cycle status. The status can be changed through REST API. Users can build an automatic status controller to mark machines as unhealthy, unreachable, retiring, or retired.

  • Disk encryption support

    To help implementing full disk encryption on client machines, sabakan accepts and stores encrypted disk encryption keys. The key can be downloaded in the next boot to decrypt disks.

    sabakan-cryptsetup is a tool for clients to encrypt disks; the tool generates a disk encryption key, encrypts it, and sends the encrypted key to sabakan. In the next boot, it downloads the encrypted key from sabakan, decrypts it, then uses it to decrypt disks.

  • Audit logs

    To track problems and life-cycle events, sabakan keeps operation logs within its etcd storage.

Programs

This repository contains these programs:

  • sabakan: the network service to manage servers.
  • sabactl: CLI tool for sabakan.
  • sabakan-cryptsetup: a utility to encrypt a block device using dm-crypt.

To see their usage, run them with -h option.

Documentation

docs directory contains tutorials and specifications.

Read getting started first.

Examples

mtest/ directory contains a set of utilities to setup sabakan on Ubuntu virtual machines.

testadata/ directory contains a sample Ignition template.

An example of production usage can be found in github.com/cybozu-go/neco. The repository bootstraps a full data center system using etcd, vault, sabakan, and many other tools.

Usage

Run sabakan with docker

# create directory to store OS images
$ sudo mkdir -p /var/lib/sabakan

# -advertise-url is the canonical URL of this sabakan.
$ docker run -d --read-only --cap-drop ALL --cap-add NET_BIND_SERVICE \
    --network host --name sabakan \
    --mount type=bind,source=/var/lib/sabakan,target=/var/lib/sabakan \
    quay.io/cybozu/sabakan:2.2 \
    -etcd-endpoints http://foo.bar:2379,http://zot.bar:2379 \
    -advertise-url http://12.34.56.78:10080

License

Sabakan is licensed under MIT license.

Docker images

Docker images are available on Quay.io

Owner
Cybozu Go
Go products from Cybozu
Cybozu Go
Comments
  • nilerr: internal error: nil Pkg importing

    nilerr: internal error: nil Pkg importing "github.com/onsi/gomega/types" from "github.com/onsi/gomega/internal/assertion"

    Hi.

    Caught such an error:

    $ make test -z "$(gofmt -s -l . | grep -v '^vendor' | tee /dev/stderr)" test -z "$(golint $(go list ./... | grep -v /vendor/) | grep -v '/mtest/.*: should not use dot imports' | tee /dev/stderr)" test -z "$(nilerr ./... 2>&1 | tee /dev/stderr)" nilerr: internal error: nil Pkg importing "github.com/onsi/gomega/types" from "github.com/onsi/gomega/internal/assertion" make: *** [Makefile:26: test] Error 1

    $ go version go version go1.15.2 gollvm LLVM 12.0.0git linux/amd64

    $ go env GO111MODULE="" GOARCH="amd64" GOBIN="" GOCACHE="/home/oceanfish81/.cache/go-build" GOENV="/home/oceanfish81/.config/go/env" GOEXE="" GOFLAGS="" GOHOSTARCH="amd64" GOHOSTOS="linux" GOINSECURE="" GOMODCACHE="/home/oceanfish81/go/pkg/mod" GONOPROXY="" GONOSUMDB="" GOOS="linux" GOPATH="/home/oceanfish81/go" GOPRIVATE="" GOPROXY="https://proxy.golang.org,direct" GOROOT="/home/oceanfish81/gollvm_dist" GOSUMDB="sum.golang.org" GOTMPDIR="" GOTOOLDIR="/home/oceanfish81/gollvm_dist/tools" GCCGO="/home/oceanfish81/gollvm_dist/bin/llvm-goc" AR="ar" CC="/usr/bin/clang" CXX="/usr/bin/clang++" CGO_ENABLED="1" GOMOD="/home/oceanfish81/sabakan/go.mod" CGO_CFLAGS="-g -O2" CGO_CPPFLAGS="" CGO_CXXFLAGS="-g -O2" CGO_FFLAGS="-g -O2" CGO_LDFLAGS="-g -O2" PKG_CONFIG="pkg-config" GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build032338975=/tmp/go-build -gno-record-gcc-switches -funwind-tables"

    CC @tenntenn

  • Add kernel parameters support

    Add kernel parameters support

    ハードコードされているiPXE scriptのKernel parameterを変更できるようにしました。

    • sabakanにkernel paramsを登録・取得するためのAPIを追加
    • etcdにkernel paramsを登録・取得するための処理を追加
    • etcdにkernel paramsのデータスキーマを仕様を決定
    • sabactlにkernel paramsを登録・取得するコマンドを追加
    • iPXEブート時に登録されたkernel paramsを含めてiPXE scriptの生成
  • Fix remaining deleted image bug

    Fix remaining deleted image bug

    When deleting an image by sabactl images delete, the images are remaining on the local disk.

    $ sabactl images upload 100.000 kernel initrd
    $ sabactl images delete 100.000
    $ sabactl images index
    [
      {
        "id": "1745.5.0",
        "date": "2018-07-17T22:02:03.097669114Z",
        "urls": [
          "http://10.69.0.3:10080/api/v1/images/coreos/1745.5.0",
          "http://10.69.0.195:10080/api/v1/images/coreos/1745.5.0",
          "http://10.69.1.131:10080/api/v1/images/coreos/1745.5.0"
        ],
        "exists": true
      }
    ]
    $ sudo ls /var/lib/sabakan/images/coreos/
    100.000  1745.5.0
    
  • Add /api/v1/machines

    Add /api/v1/machines

    Add POST,PUT,GET /api/v1/machines

    • index.go: Indexing structure for search by GET.
    • query.go: Query collection using etcd and index.
    • Add EtcdWatcher() which execute goroutine to watch etcd. When KV is updated, It will update index.
  • sabactl remote-config

    sabactl remote-config

    This changes implements sub-commands of sabactl as the following:

    $ sabactl remote-config get
    $ sabactl remote-config set -f <config.json>
    

    Remote config is described in JSON:

    {
      "node-ipv4-offset": "10.0.0.0",
      "node-rack-shift": 5,
      "bmc-ipv4-offset": "10.1.0.0",
      "bmc-rack-shift": 2,
      "node-ip-per-node": 3,
      "bmc-ip-per-node": 1
    }
    
  • Examples set auto fields but those values are overridden by sabakan

    Examples set auto fields but those values are overridden by sabakan

    Describe the bug

    The examples at:

    https://github.com/cybozu-go/sabakan/blob/main/docs/getting_started.md#register-machines https://github.com/cybozu-go/sabakan/blob/c77d1989472bc72ceb47b4b1620152a6b3e455fb/docs/machine.md#machine-struct

    Set auto fields like bmc.ipv4 but those are overridden by sabakan, so I think those fields should be removed from the examples.

    Also the documentation should explicitly mention that auto fields are overridden and must not be set by the user. I think the Values for auto fields are filled by sabakan at registration. phrase is not explicit enough, because its ambiguous whether they can/should be set afterwards.

  • Error: Bad Request: invalid request: labels contain invalid character

    Error: Bad Request: invalid request: labels contain invalid character

    Describe the bug

    While trying to follow the Getting Started / Register Machines procedure at https://github.com/cybozu-go/sabakan/blob/main/docs/getting_started.md#register-machines, it fails with:

    Error: Bad Request: invalid request: labels contain invalid character
    

    This is due to the space character in the labels.product machine definition:

    [
      {
        "labels": {
          "product": "Dell R640"
        },
    

    I think the code or documentation should be updated to prevent this error.

    PS: there's a space at https://github.com/cybozu-go/sabakan/blob/c77d1989472bc72ceb47b4b1620152a6b3e455fb/docs/machine.md#machine-struct too

  • Update for etcd 3.4.16

    Update for etcd 3.4.16

    • Update for etcd 3.4.16 Use etcdutil v1.4.0. Import etcd 3.4.16 using corresponding commit hash. Import from "go.etcd.io". CI downloads etcd images from quay.io/coreos/etcd instead of quay.io/cybozu/etcd, because it removes unnecessary dependency.

    • Update packages Update imported packages except for the following ones:

      • github.com/coreos/ignition v0.35 -> v2.10.1 (I suspect some reason not to upgrade it)
      • go.universe.tf/netboot (No release is provided and the latest commit fails at its CI)

    Signed-off-by: Daichi Sakaue [email protected]

  • Upgrade CT version

    Upgrade CT version

    Fix CT version. Because It was downgraded by the following PRs.

    https://github.com/cybozu-go/sabakan/pull/195 https://github.com/cybozu-go/sabakan/pull/196

    Signed-off-by: Masayuki Ishii [email protected]

  • [sabakan-cryptsetup] support multiple ciphers

    [sabakan-cryptsetup] support multiple ciphers

    This PR revamps sabakan-cryptsetup to:

    • Auto detect physical disks to be encrypted,
    • Document formal specifications, and
    • Support cipher and key size selection

    Additionally, this PR adds a new API at /api/v1/cryptsetup to download sabakan-cryptsetup. The location of sabakan-cryptsetup is specified by SABAKAN_CRYPTSETUP environment variable.

    If SABAKAN_CRYPTSETUP is not set, sabakan looks for sabakan-cryptsetup in the same directory where sabakan exists.

  • Revamp Ignition template

    Revamp Ignition template

    This PR revamps ignition template implementations.

    Changes in REST APIs and command-line usage break backward-compatibility. Changes in template syntax do not break backward-compatibility. Template data in etcd are converted automatically. The etcd schema version is bumped to 3.

    One most notable change is that sabakan can now handle multiple Ignition specification versions. To define an ignition template for spec version 2.3.0, specify version in template YAML as follows:

    version: 2.3
    passwd: passwd.yml
    files:
      - /etc/hostname
    ...
    
  • Support Ignition v3

    Support Ignition v3

    What

    From the Flatcar Container Linux 3.1.85.0.0, Ignition v3 is supported. https://www.flatcar.org/docs/latest/provisioning/ignition/specification/#ignition-v3

    How

    Describe how to address the issue.

    Checklist

    • [ ] Finish implentation of the issue
    • [ ] Test all functions
    • [ ] Have enough logs to trace activities
    • [ ] Notify developers of necessary actions
Modern network boot server.
Modern network boot server.

bofied demo.mp4 Modern network boot server. Overview bofied is a network boot server. It provides everything you need to PXE boot a node, from a (prox

Dec 17, 2022
Ugg boot is a tool for people wanting to have some comfort in their lives.

Ugg Boot Ugg boot is a tool for people wanting to have some comfort in their lives. It provides a simple way to update Go executables and list availab

Aug 28, 2022
Echo-server - An HTTP echo server designed for testing applications and proxies

echo-server An HTTP echo server designed for testing applications and proxies. R

Dec 20, 2022
Designed to support DNS brute-forcing with a minimal number of network connections

Fast Use of DNS Resolvers Designed to support DNS brute-forcing with a minimal number of network connections. Installation go get -v -u github.com/caf

Dec 8, 2022
Capdns is a network capture utility designed specifically for DNS traffic. This utility is based on tcpdump.
Capdns is a network capture utility designed specifically for DNS traffic. This utility is based on tcpdump.

Capdns is a network capture utility designed specifically for DNS traffic. This utility is based on tcpdump. Some of its features include: Unde

Feb 26, 2022
TFTP and HTTP server specifically designed to serve iPXE ROMs and scripts.
TFTP and HTTP server specifically designed to serve iPXE ROMs and scripts.

pixie TFTP and HTTP server specifically designed to serve iPXE ROMs and scripts. pixie comes embedded with the following ROMs provided by the iPXE pro

Dec 31, 2022
Package socket provides a low-level network connection type which integrates with Go's runtime network poller to provide asynchronous I/O and deadline support. MIT Licensed.

socket Package socket provides a low-level network connection type which integrates with Go's runtime network poller to provide asynchronous I/O and d

Dec 14, 2022
Magma is an open-source software platform that gives network operators an open, flexible and extendable mobile core network solution.
Magma is an open-source software platform that gives network operators an open, flexible and extendable mobile core network solution.

Connecting the Next Billion People Magma is an open-source software platform that gives network operators an open, flexible and extendable mobile core

Dec 31, 2022
Optimize Windows's network/NIC driver settings for NewTek's NDI(Network-Device-Interface).

windows-ndi-optimizer[WIP] Optimize Windows's network/NIC driver settings for NewTek's NDI(Network-Device-Interface). How it works This is batchfile d

Apr 15, 2022
A simple network analyzer that capture http network traffic
A simple network analyzer that capture http network traffic

httpcap A simple network analyzer that captures http network traffic. support Windows/MacOS/Linux/OpenWrt(x64) https only capture clienthello colorful

Oct 25, 2022
Zero Trust Network Communication Sentinel provides peer-to-peer, multi-protocol, automatic networking, cross-CDN and other features for network communication.
Zero Trust Network Communication Sentinel provides peer-to-peer, multi-protocol, automatic networking, cross-CDN and other features for network communication.

Thank you for your interest in ZASentinel ZASentinel helps organizations improve information security by providing a better and simpler way to protect

Nov 1, 2022
Broadcast-server - A simple Go server that broadcasts any data/stream

broadcast A simple Go server that broadcasts any data/stream usage data You can

Oct 21, 2022
Package raw enables reading and writing data at the device driver level for a network interface. MIT Licensed.

raw Package raw enables reading and writing data at the device driver level for a network interface. MIT Licensed. For more information about using ra

Dec 28, 2022
Node for providing data into Orakuru network

Orakuru's crystal-ball Node for providing data into Orakuru network. Configuration Crystal-ball uses environment variables and configuration files for

Jan 20, 2022
Golang Client for querying Tor network data using the Onionoo service.

gonion Lightweight Golang wrapper for querying Tor network data using the Onionoo service. package main import ( "github.com/R4yGM/gonion"

May 11, 2022
GO2P is a P2P framework, designed with flexibility and simplicity in mind
GO2P is a P2P framework, designed with flexibility and simplicity in mind

go2p golang p2p framework By v-braun - viktor-braun.de. Description GO2P is a P2P framework, designed with flexibility and simplicity in mind. You can

Jan 5, 2023
Subfinder is a subdomain discovery tool that discovers valid subdomains for websites. Designed as a passive framework to be useful for bug bounties and safe for penetration testing.
Subfinder is a subdomain discovery tool that discovers valid subdomains for websites. Designed as a passive framework to be useful for bug bounties and safe for penetration testing.

Fast passive subdomain enumeration tool. Features • Install • Usage • API Setup • License • Join Discord Subfinder is a subdomain discovery tool that

Jan 4, 2023
httpx is a fast and multi-purpose HTTP toolkit allows to run multiple probers using retryablehttp library, it is designed to maintain the result reliability with increased threads.
httpx is a fast and multi-purpose HTTP toolkit allows to run multiple probers using retryablehttp library, it is designed to maintain the result reliability with increased threads.

Features • Installation • Usage • Running httpx • Notes • Join Discord httpx is a fast and multi-purpose HTTP toolkit allow to run multiple probers us

Jan 8, 2023
A vote botting wrapper for GoLang designed for Minecraft: Pocket Servers.

libvote A vote botting wrapper for GoLang designed for Minecraft: Pocket Servers by Jviguy and JustTal. Disclaimer Usage of libvote requires your own

Apr 17, 2022