Lagoon - Simple Linux package repository mirror

Lagoon - Simple Linux package repository mirror

A lagoon is a shallow stretch of water separated from the sea by a reef or sandbank. Lagoon can be used to mirror package repositories or parts of these repositories from the internet hence the name Lagoon.

When running Linux servers in an enterprise environment it is useful to have all servers running the same versions of software. In order to accomplish this, package repositories must be frozen at a certain point in time because normal (public) repositories are constantly updated and 'on the move'.

Lagoon can be used to set up a local mirror for upstream OS package repositories and makes it possible to capture certain points in time or so called snapshots. It also provides the latest snapshot or version of the upstream repository. Lagoon is only capable of providing very basic functionality and is not a replacement for Red Hat's Satellite, Foreman's Katello or Pulp.

Lagoon synchronizes with the remote repository and stores the files in the upstream folder. When synchronization is complete a point in time snapshot is made in the staging folder according to the following pattern 20060102 (therefore the snapshot resolution cannot be smaller than a day). After the point in time snapshot is created, it is published in the public folder using the same pattern. The last snapshot is also published as latest.

Running Lagoon

Lagoon can be run as a standalone golang binary or from a Docker container. A Docker Compose file is included for reference.

Run Lagoon from a Docker container:

docker run --name lagoon -v $PWD/lagoon.yml:/etc/lagoon/lagoon.yml lagoon:latest

Requirements

The following dependencies are needed for running Lagoon:

  • rsync
  • yum-utils
  • createrepo

Supported synchronisation methods

Sync method Supported Status
Rsync yes
RPM reposync beta Basic sync. TODO: implement errata support

File storage

The treeview below shows how snapshots are stored, for example repo1 consists of daily snapshots and repo2 consists of weekly snapshots. Each folder contains the files which were downloaded from the upstream repository at that certain moment in time. Hardlinks are being used for efficient storage and to make sure files do not disappear from a staging snapshot when they are deleted from the upstream content. Staging snapshots are published with symlinks, the latest snapshot always points to the last staging snapshot.

/var/lib/lagoon/
|-- public
|   |-- repo1
|   |   |-- 20220126 -> /var/lib/lagoon/staging/repo1/20220126
|   |   |-- 20220127 -> /var/lib/lagoon/staging/repo1/20220127
|   |   |-- 20220128 -> /var/lib/lagoon/staging/repo1/20220128
|   |   |-- 20220129 -> /var/lib/lagoon/staging/repo1/20220129
|   |   |-- 20220130 -> /var/lib/lagoon/staging/repo1/20220130
|   |   `-- latest -> /var/lib/lagoon/staging/repo1/20220130
|   `-- repo2
|       |-- 20220115 -> /var/lib/lagoon/staging/repo2/20220115
|       |-- 20220122 -> /var/lib/lagoon/staging/repo2/20220122
|       |-- 20220129 -> /var/lib/lagoon/staging/repo2/20220129
|       `-- latest -> /var/lib/lagoon/staging/repo2/20220129
|-- staging
|   |-- repo1
|   |   |-- 20220126
|   |   |-- 20220127
|   |   |-- 20220128
|   |   |-- 20220129
|   |   `-- 20220130
|   `-- repo2
|       |-- 20220115
|       |-- 20220122
|       `-- 20220129
`-- upstream
    |-- repo1
    |   |-- file_1
    |   `-- file_n
    `-- repo2
        |-- file_1
        `-- file_n

Lagoon can also take care of automatically freeing up diskspace by removing snapshots which aren't used anymore. This can be configured by telling Lagoon how much snapshots it has to keep for a certain repository.

Configuration

See lagoon.example.yml for example configuration.

repositories:
  - id: docker-ce_centos-7 # Unique id
    # Name of the repo
    name: Docker CE CentOS 7 x86_64
    # Type of remote repository (rsync or reposync)
    type: reposync
    # Upstream rsync url or reposync multiline string with yum repo config
    src: |
      [docker-ce-stable-centos7]
      baseurl = https://download.docker.com/linux/centos/7/x86_64/stable
      enabled = 1
      gpgcheck = 1
      gpgkey = https://download.docker.com/linux/centos/gpg
      name = Docker CE Stable - x86_64
    # Destination the repo (absolute path)
    dest: /var/lib/lagoon
    # Cron sync expression see: https://github.com/robfig/cron
    cron: "*/10 * * * * *"
    # Number of snapshots to keep
    snapshots: 52
    # List of directories to exclude from rsync
    #exclude: []

Logging and monitoring

By default Lagoon logs to stdout using JSON format. In order to enable debug logging start Lagoon with -d parameter. For human readable logging start Lagoon with -h parameter. Each separate sync job can be identified by the repository name and a unique ID.

Lagoon can be monitored with Prometheus and exposes its metrics on port 9000 at /metrics. In addition to the standard golang metrics the following Lagoon specific metrics are exposed:

Metric Description
lagoon_sync_total The total number of repo syncs
lagoon_sync_duration_seconds The sync duration

Building Lagoon

We will use Docker as a build environment so local installation of build tools is not required. Execute the commands from the root of the project.

Building a Docker image:

docker build -f build/Dockerfile -t lagoon .

You can either run a Docker container from the image or use the following command to extract the lagoon binary to run it separately:

docker run --rm -v $PWD:/build lagoon cp /etc/lagoon/lagoon /build

Developing Lagoon

The preferred method for developing Lagoon is using VSCode with the 'Remote - Containers' extension ms-vscode-remote.remote-containers. A .devcontainer context is included with the project in order to open it in a remote container and get your development environment up-and-running quickly.

Owner
Klaas Jan Dijksterhuis
Klaas Jan Dijksterhuis
Similar Resources

Linux provisioning scripts + application deployment tools. Suitable for self-hosting and hobby-scale application deployments.

Apollo Linux provisioning scripts + application deployment tools. Suitable for self-hosting and hobby-scale application deployments. Philosophy Linux-

Feb 7, 2022

RancherOS v2 is an immutable Linux distribution built to run Rancher and it's corresponding Kubernetes distributions RKE2 and k3s

RancherOS v2 is an immutable Linux distribution built to run Rancher and it's corresponding Kubernetes distributions RKE2 and k3s. It is built using the cOS-toolkit and based on openSUSE

May 10, 2022

A demo repository that shows CI/CD integration using DroneCI + ArgoCD + Kubernetes.

A demo repository that shows CI/CD integration using DroneCI + ArgoCD + Kubernetes.

CI/CD Demo This is the demo repo for my blog post. This tutorial shows how to build CI/CD pipeline with DroneCI and ArgoCD. In this demo, we use Drone

Apr 26, 2022

A template repository to quickly scaffold a Kubewarden policy written with Go language

go-policy-template This is a template repository that can be used to to quickly scaffold a Kubewarden policy written with Go language. Don't forget to

Apr 29, 2022

ArgoCD is widely used for enabling CD GitOps. ArgoCD internally builds manifest from source data in Git repository, and auto-sync it with target clusters.

ArgoCD is widely used for enabling CD GitOps. ArgoCD internally builds manifest from source data in Git repository, and auto-sync it with target clusters.

ArgoCD Interlace ArgoCD is widely used for enabling CD GitOps. ArgoCD internally builds manifest from source data in Git repository, and auto-sync it

May 7, 2022

This repository contains Prowjob configurations for Amazon EKS Anywhere.

Amazon EKS Anywhere Prow Jobs This repository contains Prowjob configuration for the Amazon EKS Anywhere project, which includes the eks-anywhere and

Apr 18, 2022

Synchronise a directory's contents with a git repository.

git-volume-reloader Synchronise a directory's contents with a git repository. Synchronisation is triggered by a webhook sent by the git service provid

Mar 20, 2022

Helmsman repository management tool

Richman A tool to manage helmsman specification toml files. Install go get -u github.com/kronostechnologies/richman # OR go install Usage Update char

Oct 7, 2021

A tool that allows you to manage Kubernetes manifests for your services in a Git repository

kuberpult Readme for users About Kuberpult is a tool that allows you to manage Kubernetes manifests for your services in a Git repository and manage t

Apr 7, 2022
Comments
  • chore(deps): bump golang from 1.18.1-buster to 1.18.2-buster in /.devcontainer

    chore(deps): bump golang from 1.18.1-buster to 1.18.2-buster in /.devcontainer

    Bumps golang from 1.18.1-buster to 1.18.2-buster.

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • chore(deps): bump golang from 1.18.1-buster to 1.18.2-buster in /build

    chore(deps): bump golang from 1.18.1-buster to 1.18.2-buster in /build

    ⚠️ Dependabot is rebasing this PR ⚠️

    Rebasing might not happen immediately, so don't worry if this takes some time.

    Note: if you make any changes to this PR yourself, they will take precedence over the rebase.


    Bumps golang from 1.18.1-buster to 1.18.2-buster.

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
k8s-image-swapper Mirror images into your own registry and swap image references automatically.
k8s-image-swapper Mirror images into your own registry and swap image references automatically.

k8s-image-swapper Mirror images into your own registry and swap image references automatically. k8s-image-swapper is a mutating webhook for Kubernetes

May 1, 2022
Dgraph Backup and Restore (cloud). Read-only mirror.

dgbrx Dgraph Backup and Restore X dgbrx is a Go commandline tool which helps to do a backup, restore or clean on a Dgraph Cloud (aka slash / managed)

Oct 28, 2021
Defalyzer: A cross-platform software that focuses on defacement analyzing and mirror tracking
Defalyzer: A cross-platform software that focuses on defacement analyzing and mirror tracking

Project Defalyzer is a cross-platform software that focuses on defacement analyz

Apr 28, 2022
Linux Controllers for Kubernetes

Tambourine Kubelet replacement with Built in Linux extensions Development Success: Install, Manage, and Observe a new systemd service from Kubernetes.

Jun 2, 2021
A Go library for the Linux Landlock sandboxing feature

Go landlock library The Go landlock library provides an interface to Linux 5.13's Landlock kernel sandboxing features. The library provides access to

Mar 30, 2022
Sysctl.conf and other system tunings for Linux

PATCHFILES I came to the idea to create patchfiles, when I saw lots of config files people create. Patchfiles implements various config scripts into o

Apr 7, 2022
Linux Gestures
 Linux Gestures

Swipe Gestures on Linux. https://evuraan.info/Swipe/ Screengrab: https://evuraan.info/evuraan/stuff/Swipe.mp4 Features Swipe uses a novel yet simple c

Apr 29, 2022
A Rancher and Kubernetes optimized immutable Linux distribution based on openSUSE

RancherOS v2 WORK IN PROGRESS RancherOS v2 is an immutable Linux distribution built to run Rancher and it's corresponding Kubernetes distributions RKE

Apr 20, 2022
Truly Minimal Linux Distribution for Containers

Statesman Statesman is a minimal Linux distribution, running from memory, that has just enough functionality to run OCI-compatible containers. Rationa

Nov 12, 2021
🔥 🔥 Open source cloud native security observability platform. Linux, K8s, AWS Fargate and more. 🔥 🔥
🔥 🔥   Open source cloud native security observability platform. Linux, K8s, AWS Fargate and more. 🔥 🔥

CVE-2021-44228 Log4J Vulnerability can be detected at runtime and attack paths can be visualized by ThreatMapper. Live demo of Log4J Vulnerability her

May 10, 2022