Argo-CD Autopilot

Argo Logo

Codefresh build status codecov Documentation Status slack

Introduction

New users to GitOps and Argo CD are not often sure how they should structure their repos, add applications, promote apps across environments, and manage the Argo CD installation itself using GitOps.

Argo CD Autopilot saves operators time by:

  • Installing and managing the Argo CD application using GitOps.
  • Providing a clear structure for how applications are to be added and updated, all from git.
  • Creating a simple pattern for making updates to applications and promoting those changes across environments.
  • Enabling better disaster recovery by being able to bootstrap new clusters with all the applications previously installed.
  • Handles secrets for Argo CD to prevent them from spilling into plaintext git. (Soon to come)

The Argo-CD Autopilot is a tool which offers an opinionated way of installing Argo-CD and managing GitOps repositories.

Installation

Using brew:

# install
brew install argocd-autopilot

# check the installation
argocd-autopilot version

Linux and WSL (using curl):

# get the latest version or change to a specific version
VERSION=$(curl --silent "https://api.github.com/repos/argoproj-labs/argocd-autopilot/releases/latest" | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/')

# download and extract the binary
curl -L --output - https://github.com/argoproj-labs/argocd-autopilot/releases/download/$VERSION/argocd-autopilot-linux-amd64.tar.gz | tar zx

# move the binary to your $PATH
mv ./argocd-autopilot-* /usr/local/bin/argocd-autopilot

# check the installation
argocd-autopilot version

Mac (using curl):

# get the latest version or change to a specific version
VERSION=$(curl --silent "https://api.github.com/repos/argoproj-labs/argocd-autopilot/releases/latest" | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/')

# download and extract the binary
curl -L --output - https://github.com/argoproj-labs/argocd-autopilot/releases/download/$VERSION/argocd-autopilot-darwin-amd64.tar.gz | tar zx

# move the binary to your $PATH
mv ./argocd-autopilot-* /usr/local/bin/argocd-autopilot

# check the installation
argocd-autopilot version

Docker

When using the Docker image, you have to provide the .kube and .gitconfig directories as mounts to the running container:

docker run \
  -v ~/.kube:/home/autopilot/.kube \
  -v ~/.gitconfig:/home/autopilot/.gitconfig \
  -it quay.io/argoprojlabs/argocd-autopilot <cmd> <flags>

Getting Started

# All of the commands need your git token with the --git-token flag,
# or the GIT_TOKEN env variable:

    export GIT_TOKEN=<YOUR_TOKEN>

# The commands will also need your repo clone URL with the --repo flag,
# or the GIT_REPO env variable:

    export GIT_REPO=<REPO_URL>

# 1. Run the bootstrap installation on your current kubernetes context.
# This will install argo-cd as well as the application-set controller.

    argocd-autopilot repo bootstrap

# Please note that this will automatically attempt to create a private repository,
# if the clone URL references a non-existing one. If the repository already exists,
# the command will just clone it.

# 2. Create your first project

    argocd-autopilot project create my-project

# 3. Install your first application on your project

    argocd-autopilot app create demoapp --app github.com/argoproj-labs/argocd-autopilot/examples/demo-app/ -p my-project

Now, if you go to your Argo-CD UI, you should see something similar to this:

Head over to our Getting Started guide for further details.

How it works

The autopilot bootstrap command will deploy an Argo-CD manifest to a target k8s cluster, and will commit an Argo-CD Application manifest under a specific directory in your GitOps repository. This Application will manage the Argo-CD installation itself - so after running this command, you will have an Argo-CD deployment that manages itself through GitOps.

From that point on, the user can create Projects and Applications that belong to them. Autopilot will commit the required manifests to the repository. Once committed, Argo-CD will do its magic and apply the Applications to the cluster.

An application can be added to a project from a public git repo + path, or from a directory in the local filesystem.

Architecture

Argo-CD Autopilot Architecture

Autopilot communicates with the cluster directly only during the bootstrap phase, when it deploys Argo-CD. After that, most commands will only require access to the GitOps repository. When adding a Project or Application to a remote k8s cluster, autopilot will require access to the Argo-CD server.

You can read more about it in the official proposal doc.

Features

  • Opinionated way to build a multi-project multi-application system, using GitOps principles.
  • Create a new GitOps repository, or use an existing one.
  • Supports creating the entire directory structure under any path the user requires.
  • When adding applications from a public repo, allow committing as either a kustomization that references the public repo, or as a "flat" manifest file containing all the required resources.
  • Use a different cluster from the one Argo-CD is running on, as a default cluster for a Project, or a target cluster for a specific Application.

Development Status

Argo-CD autopilot is currently under active development. Some of the basic commands are not yet implemented, but we hope to complete most of them in the coming weeks.

Slack Channel

Join us in channel #argo-autopilot in CNCF slack workspace.

Click here to join: https://slack.cncf.io/

Comments
  • Fix cli-local target for Homebrew support

    Fix cli-local target for Homebrew support

    I'm starting to create Homebrew formulae, thus, the following updates need to be merged to support Homebrew.

    In argocd repository, make cli-local is used by Homebrew formulae: see argocd.rb

    See current working formula using my forked repository.

    class ArgocdAutopilot < Formula
      desc "Argo-CD Autopilot"
      homepage "https://argoproj.io"
      url "https://github.com/sestegra/argocd-autopilot.git",
          tag:      "v0.1.9",
          revision: "6260363b81c3741fc33e4283f4c162af56506505"
      license "Apache-2.0"
    
      depends_on "go" => :build
    
      def install
        system "make", "cli-local"
        bin.install "dist/argocd-autopilot"
      end
    end
    
  • Do you support azure git provider ?

    Do you support azure git provider ?

    I am able to add repo using argocd but fails with argocd-autopilot FATAL authentication failed, make sure credentials are correct: GET https://dev.azure.com//api/v3/user: 401 []

  • failed to build bootstrap manifests

    failed to build bootstrap manifests

    ➜  ~ argocd-autopilot version
    v0.1.10
    

    macos (11.4) -> installed with brew

    I set GIT_TOKEN and GIT_REPO vars and run...

    ➜  ~ argocd-autopilot repo bootstrap
    

    The result is

    FATAL failed to build bootstrap manifests: accumulating resources: accumulation err='accumulating resources from 'manifests/': evalsymlink failure on '/private/var/folders/c2/t6pj5zmx11n_yqlltbdy1n9nhb8tt4/T/auto-pilot862157232/manifests' : lstat /private/var/folders/c2/t6pj5zmx11n_yqlltbdy1n9nhb8tt4/T/auto-pilot862157232/manifests: no such file or directory': evalsymlink failure on '/private/var/folders/c2/t6pj5zmx11n_yqlltbdy1n9nhb8tt4/T/auto-pilot862157232/manifests' : lstat /private/var/folders/c2/t6pj5zmx11n_yqlltbdy1n9nhb8tt4/T/auto-pilot862157232/manifests: no such file or directory
    
  • Add HA manifests

    Add HA manifests

    Using HA with Argo CD Autopilot would currently require replicating the kustomization in the manifests/base folder into another repo and losing the upstream. Adding an ha kustomization will allow users to specify a different base for Argo CD manifests and switch to an HA version.

    Ideally this would be supported with a cli flag aswell but this is a good start.

  • Bump sigs.k8s.io/kustomize/api from 0.8.8 to 0.10.0

    Bump sigs.k8s.io/kustomize/api from 0.8.8 to 0.10.0

    Bumps sigs.k8s.io/kustomize/api from 0.8.8 to 0.10.0.

    Release notes

    Sourced from sigs.k8s.io/kustomize/api's releases.

    cmd/config/v0.10.0

    16dcc98cf deprecate some cfg commands (#4048 ) 0537b59f2 support yaml formatted openapi schema (#4017)

    api/v0.10.0

    dce4ea584 Add AnchorsAweigh option to ByteReader to toggle YAML alias/anchor expansion -- this enables anchor expansion during builds by default (#4187) 67a5f6d68 support krm spec v1 and legacy path, index, and id annotations (#4190) 02cb395ec support multiple gvks in custom openapi schema (#4163) f4382738a [fix 4124] Skip local resource until all transformations have completed. (#4180) 6552b9065 Cache the OrgId for nameReferenceTransformer -- performance improvement (#4171) 2bfc7cc1b throw error instead of panic when replacements source.fieldPath doesn't exist (#4166) 0244f0919 update name references after kind change (#4165)

    cmd/config/v0.9.13

    aa92d83d8 Pin to kyaml (#4020) b465c20f6 Remove pinning to external fork 5c2c617ff fix affected cmd/config tests c8b049f57 point to natasha's fork 2e674337b expose --as-current-user via GetRunFnRunner a3ed120ef Handle errors e468d6b4d Fixing cmd/config/internal/commands/e2e/e2e_test.go 58165dfc8 comments 0e8257c38 comments 3ee157968 Fully specify KRM Functions Spec and graduate it to v1. f6e6ac032 Back to development mode; unpin the modules

    cmd/config/v0.9.12

    51605beb3 Pin to kyaml v0.10.20 3f3d3b17a Replace pkger with embed.FS compatibility 5bb736496 Fix linter error a4db686b6 Unpin everything.

    cmd/config/v0.9.11

    3a508da64 Pin to kyaml/v0.10.18 677ec868e Fix golint warning 8a529ca39 Update merge3 with deafult GVKNN matcher 7e622e138 remove go-openapi/spec,validate,strfmt from cmd/config/ 38e9c34f0 Back to dev mode; unpin the modules. 5ba45f1ef Back to mergo v0.3.5 d8015d3c9 Back to development mode; unpin the modules

    cmd/config/v0.9.10

    88911bbb6 Pin to [email protected]

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Bump github.com/spf13/cobra from 1.1.3 to 1.2.1

    Bump github.com/spf13/cobra from 1.1.3 to 1.2.1

    Bumps github.com/spf13/cobra from 1.1.3 to 1.2.1.

    Release notes

    Sourced from github.com/spf13/cobra's releases.

    v1.2.1

    Bug fixes

    • Quickfix for spf13/cobra#1437 after v1.2.0 where parallel use of the cmd.RegisterFlagCompletionFunc() (and subsequent map) now works correctly and flag completions now work again

    v1.2.0 - The completions release

    :stars: v1.2.0 - The completions release

    Welcome to v1.2.0 of Cobra! This release focuses on code completions, several critical bug fixes, some documentation updates, and security bumps. Upgrading should be simple but note please take note of the introduction of completions V2 and their default use. The v1 completions library is still available, but will be deprecated in the future. Please open an issue with any problems!


    New Features

    • Automatically adds completion command for shell completions. If a completion command is already provided, uses that instead. This will automatically provide shell completions for bash, zsh, fish, and PowerShell spf13/cobra#1192
      • Users can configure the command auto creation:
        • disable the creation of the completion command
        • disable completion descriptions
        • disable the --no-descriptions flag for "always on" completion descriptions
    • Introduction of bash completions V2, a uniform completion approach which include completion descriptions. The V1 bash completions are still available and will be deprecated in a latter release - spf13/cobra#1146
      • Note that projects providing completion through a different command name (say a command named "complete") will continue to use v1 for their own command but will also provide cobra's implicit "completion" command which will use v2, unless of course, these projects take the time to disable the default "completion" command as noted above.
    • Commands now support context being passed to completions - spf13/cobra#1265
    • Removed dependency onmitchellh/go-homedir in favor of core Go os.UserHomeDir() - https://github.com/spf13/cobra/commit/8eaca5f0f49ad747a0722d39dca7a75c34abd21a

    Bug Fixes

    • Fix trailing whitespace not being handled in powershell completion scripts spf13/cobra#1342
    • Bash completion variable leak fix spf13/cobra#1352
    • Fish shell completions correctly ignore trailing empty lines spf13/cobra#1284
    • PowerShell completions fix for "no file comp directive" - spf13/cobra#1363
    • Custom completions now correctly handle multiple shorthand flags together - spf13/cobra#1258
    • zsh completions now correctly handle ShellDirectiveCompletionNoSpace and file completion all the time - spf13/cobra#1213
    • Multiple fixes / improvements to the fish shell support - spf13/cobra#1249
    • Fix home directory config not loading correctly - spf13/cobra#1282
    • Fix for RegisterFlagCompletionFunc as a global var not working in multi-threaded programs: spf13/cobra#1423
    • Custom completions correctly do not complete flags after args when interspersed is false #1308

    Testing

    Security

    • Bump viper to 1.8.1. This corrects several issues with vulnerabilities existing in the dependency tree - spf13/cobra#1433

    Other

    Thank you to all our amazing contributors :snake::rocket:

    Changelog

    Sourced from github.com/spf13/cobra's changelog.

    Cobra Changelog

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • docs: fix edit url to use `main` branch

    docs: fix edit url to use `main` branch

    Hello 🙂, this PR fixes a broken URL of the edit button on the documentation.

    If you click the edit button on the documentation page, you will be redirected to a URL like this: https://github.com/argoproj-labs/argocd-autopilot/edit/master/docs/index.md. But we could not access the page since the branch name was renamed from master to main at some time.

    To fix the issue, we need to explicitly specify the edit_uri option of the mkdocs, which uses the master branch by default.

    I confirmed this change works on my local build by running make serve-docs command:

    Screen Shot 2022-05-07 at 20 49 15
  • Add support for bitbucket for repo bootstrap

    Add support for bitbucket for repo bootstrap

    Please add support for bitbucket for repo bootstrap & provide documentation.

    I see this issue is open https://github.com/argoproj-labs/argocd-autopilot/issues/7

    But PR https://github.com/argoproj-labs/argocd-autopilot/pull/172 is closed so not sure if bitbucket support is there but undocumented or still missing?

    Thanks

  • Bootstrap argocd installation with a helm chart

    Bootstrap argocd installation with a helm chart

    Summary

    Argocd Autopilot users could benefit from optionally specifying a helm chart for ArgoCD installation rather than just the default installation. For example, it would be nice to be able to control the bootstrap with things like initial RBAC configuration, controller replica count, or any other configuration available in the helm chart.

    Proposal

    The existing repo bootstrap should be able to install ArgoCD via a user-defined helm chart.

    For example, I have a user-defined Values.yaml file with some configuration values for rbac. I should be able to run something like argocd-autopilot repo bootstrap --helm-install=latest --values=Values.yaml

    The --helm-install flag indicates that command should use install from the helm chart at either the latest or a specified version. The --values flag is just a file location for any user defined values for the installation.

  • Bump github.com/xanzy/go-gitlab from 0.71.0 to 0.77.0

    Bump github.com/xanzy/go-gitlab from 0.71.0 to 0.77.0

    Bumps github.com/xanzy/go-gitlab from 0.71.0 to 0.77.0.

    Commits
    • 52eedaa Merge pull request #1598 from armsnyder/fix-slow-tests
    • b56a655 Merge pull request #1599 from armsnyder/deprecate-test-teardown
    • 946a151 Refactor tests to not call teardown explicitly
    • 4e7c141 Deprecate the test teardown function by moving teardown code into a test clea...
    • 65b39a6 Merge pull request #1593 from jokly/error-handler
    • 837fee3 Merge pull request #1596 from armsnyder/get-latest-pipeline
    • db9dda5 Add WithErrorHandler client option
    • 21ae872 Fix slow tests by disabling retry backoff
    • 91b78c1 New GetLatestPipeline() client method
    • 05207f5 Merge pull request #1585 from cyd01/master
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • fresh install on gitlab results in bootstrap\argo-cd: app path does not exist

    fresh install on gitlab results in bootstrap\argo-cd: app path does not exist

    Creating an empty project on gitlab and then running argocd-autopilot repo bootstrap starts the argo-cd, autopilot-boostrap and root applications, but the argo-cd application has sync status UNKNOWN with 1 error:

    ComparisonError
    rpc error: code = Unknown desc = Manifest generation error (cached): bootstrap\argo-cd: app path does not exist
    

    The bootstrap\argo-cd path exists in the repo!

  • when runtime repo has branch protection, bootstrap fails too late

    when runtime repo has branch protection, bootstrap fails too late

    if the "push" fails due to branch protection issues, the process fails after already applying argo-cd to the cluster. a better solution would be to make some dummy push early on (before applying anything), just to make sure that everything will work well later on

  • failure to add an app on a remote-cluster with a long name

    failure to add an app on a remote-cluster with a long name

    when cluster-name + app-name are too long (> 63 characters), the resulting app name is not a valid k8s label value. this causes the app to fail to not be able to apply successfully to the cluster.

Notifications for Argo CD
Notifications for Argo CD

Argo CD Notifications Argo CD Notifications continuously monitors Argo CD applications and provides a flexible way to notify users about important cha

Nov 27, 2022
Argo CD ApplicationSet Controller

The ApplicationSet controller manages multiple Argo CD Applications as a single ApplicationSet unit, supporting deployments to large numbers of clusters, deployments of large monorepos, and enabling secure Application self-service.

Dec 14, 2022
Automatic container image update for Argo CD

Argo CD Image Updater Introduction Argo CD Image Updater is a tool to automatically update the container images of Kubernetes workloads which are mana

Dec 25, 2022
A Kubernetes operator for managing Argo CD clusters.

Argo CD Operator A Kubernetes operator for managing Argo CD clusters. Documentation See the documentation for installation and usage of the operator.

Dec 14, 2022
Support for extending Argo CD

Argo CD Extensions To enable Extensions for your Argo CD cluster will require just a single kubectl apply. Here we provide a way to extend Argo CD suc

Dec 20, 2022
Hera is a Python framework for constructing and submitting Argo Workflows.

Hera is an Argo Workflows Python SDK. Hera aims to make workflow construction and submission easy and accessible to everyone! Hera abstracts away workflow setup details while still maintaining a consistent vocabulary with Argo Workflows.

Dec 31, 2022
A series of controllers for configuring namespaces to accomodate Argo

argo-controller A series of controllers for configuring namespaces to accomodate Argo. ArgoCD TBD Argo Workflows Make a service account in every names

Jan 4, 2022
Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes.
Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes.

What is Argo Workflows? Argo Workflows is an open source container-native workflow engine for orchestrating parallel jobs on Kubernetes. Argo Workflow

Dec 10, 2021
Argo Rollout visualization in Argo CD Web UI
Argo Rollout visualization in Argo CD Web UI

Rollout Extension The project introduces the Argo Rollout dashboard into the Argo CD Web UI. Quick Start Install Argo CD and Argo CD Extensions Contro

Dec 29, 2022
The MySQL Cluster Autopilot Management with GTID and Raft
The MySQL Cluster Autopilot Management with GTID and Raft

Xenon Overview Xenon is a MySQL HA and Replication Management tool using Raft protocol. Xenon has many cool features, such as: Fast Failover with no l

Jan 3, 2023
Autopilot in Go for docking the SpaceX Dragon capsule in the simulator
Autopilot in Go for docking the SpaceX Dragon capsule in the simulator

Dragon ISS Docking Autopilot in Go Autopilot written in Go and executed as WebAssembly for docking the SpaceX Dragon capsule in the official simulator

Sep 11, 2022
Automating Kubernetes Rollouts with Argo and Prometheus. Checkout the demo URL below
Automating Kubernetes Rollouts with Argo and Prometheus. Checkout the demo URL below

observe-argo-rollout Demo for Automating and Monitoring Kubernetes Rollouts with Argo and Prometheus Performing Demo The demo can be found on Katacoda

Nov 16, 2022
A plugin for argo which behaves like I'd like

argocd-lovely-plugin An ArgoCD plugin to perform various manipulations in a sensible order to ultimately output YAML for Argo CD to put into your clus

Dec 27, 2022
Simple example using Git actions + Argo CD + K8S + Docker and GO lang

CICD-simple_example Simple example using Git actions + Argo CD + K8S + Docker and GO lang Intro Pre reqs Have an ArgoCD account and Installed. Docker

Oct 28, 2021
Notifications for Argo CD
Notifications for Argo CD

Argo CD Notifications Argo CD Notifications continuously monitors Argo CD applications and provides a flexible way to notify users about important cha

Nov 27, 2022
Argo CD ApplicationSet Controller

The ApplicationSet controller manages multiple Argo CD Applications as a single ApplicationSet unit, supporting deployments to large numbers of clusters, deployments of large monorepos, and enabling secure Application self-service.

Dec 14, 2022
Automatic container image update for Argo CD

Argo CD Image Updater Introduction Argo CD Image Updater is a tool to automatically update the container images of Kubernetes workloads which are mana

Dec 25, 2022
A Kubernetes operator for managing Argo CD clusters.

Argo CD Operator A Kubernetes operator for managing Argo CD clusters. Documentation See the documentation for installation and usage of the operator.

Dec 14, 2022
Support for extending Argo CD

Argo CD Extensions To enable Extensions for your Argo CD cluster will require just a single kubectl apply. Here we provide a way to extend Argo CD suc

Dec 20, 2022
Hera is a Python framework for constructing and submitting Argo Workflows.

Hera is an Argo Workflows Python SDK. Hera aims to make workflow construction and submission easy and accessible to everyone! Hera abstracts away workflow setup details while still maintaining a consistent vocabulary with Argo Workflows.

Dec 31, 2022