Kubedd – Check migration issues of Kubernetes Objects while K8s upgrade

Kubedd

Motivation

Currently there is no easy way to upgrade kubernetes objects in case of kubernetes upgrade. There are some tools which are available for this purpose, but we found then inadequate for migration requirements.

kubedd is a tool to check issues in migration of kubernetes yaml objects from one kubernetes version to another.

It uses openapi spec provided by the kubernetes with releases, for eg. in case of target kubernetes version 1.22 openapi spec for 1.22, to validate the kubernetes objects for depreciation or non-conformity with openapi spec.

Supported input formats

  1. Directory containing files to be validated
  2. Read kubernetes objects directly from cluster.Uses kubectl.kubernetes.io/last-applied-configuration to get last applied configuration and in its absence uses the manifest itself.

It provides details of issues with the kubernetes object in case they are migrated to cluster with newer kubernetes version.

Install

Download kubedd, and it is ready for use.

Usage

./kubedd 

Results for cluster at version 1.12 to 1.22
-------------------------------------------
>>>> Removed API Version's <<<<
 Namespace   Name                          Kind         API Version (Current Available)   Replace With API Version (Latest Available)   Migration Status                               
 prod        demmoo-prod-ingress           Ingress      extensions/v1beta1                                                              Alert! cannot migrate kubernetes version  
 prod        devtron-static-prod-ingress   Ingress      extensions/v1beta1                                                              Alert! cannot migrate kubernetes version  
 prod        ghost-blog-dt-prod            Ingress      extensions/v1beta1                                                              Alert! cannot migrate kubernetes version  
 prod        ghost-blog-dt-prod-auth       Ingress      extensions/v1beta1                                                              Alert! cannot migrate kubernetes version  
 prod        ghost-devtron-blog-prod       Ingress      extensions/v1beta1                                                              Alert! cannot migrate kubernetes version  
 prod        oauth2-proxy                  Ingress      extensions/v1beta1                                                              Alert! cannot migrate kubernetes version  
 prod        telemetry-prod-ingress        Ingress      extensions/v1beta1                                                              Alert! cannot migrate kubernetes version  
 prod        ghost-blog-dt-prod            Deployment   extensions/v1beta1                apps/v1                                       can be migrated with just apiVersion change    
 prod        ghost-devtron-blog-prod       Deployment   extensions/v1beta1                apps/v1                                       can be migrated with just apiVersion change  

Arguments

./kubedd --help
Validates migration of Kubernestes YAML file against specific kubernetes version, It provides details of issues with the kubernetes object in case they are migrated to cluster with newer kubernetes version

Usage:
  kubedd  [file...] [flags]

Flags:
  -d, --directories strings                   A comma-separated list of directories to recursively search for YAML documents
      --force-color                           Force colored output even if stdout is not a TTY
  -h, --help                                  help for kubedd
      --ignore-keys-for-deprecation strings   A comma-separated list of keys to be ignored for depreciation check (default [metadata*,status*])
      --ignore-keys-for-validation strings    A comma-separated list of keys to be ignored for validation check (default [status*,metadata*])
      --ignore-kinds strings                  A comma-separated list of kinds to be skipped (default [event,CustomResourceDefinition])
      --ignore-namespaces strings             A comma-separated list of namespaces to be skipped (default [kube-system])
      --ignore-null-errors                    Ignore null value errors (default true)
      --ignored-filename-patterns strings     An alias for ignored-path-patterns
  -i, --ignored-path-patterns strings         A comma-separated list of regular expressions specifying paths to ignore
      --insecure-skip-tls-verify              If true, the server's certificate will not be checked for validity. This will make your HTTPS connections insecure
      --kubeconfig string                     Path of kubeconfig file of cluster to be scanned
      --kubecontext string                    Kubecontext to be selected
      --select-kinds strings                  A comma-separated list of kinds to be selected, if left empty all kinds are selected
      --select-namespaces strings             A comma-separated list of namespaces to be selected, if left empty all namespaces are selected
      --source-kubernetes-version string      Version of Kubernetes of the cluster on which kubernetes objects are deployed currently, ignored in case cluster is provided. In case of directory defaults to same as target-kubernetes-version.
      --source-schema-location string         SourceSchemaLocation is the file path of kubernetes versions of the cluster on which manifests are deployed. Use this in air-gapped environment where it internet access is unavailable.
      --target-kubernetes-version string      Version of Kubernetes to migrate to eg 1.22, 1.21, 1.12 (default "1.22")
      --target-schema-location string         TargetSchemaLocation is the file path of kubernetes version of the target cluster for these manifests. Use this in air-gapped environment where it internet access is unavailable.
      --version                               version for kubedd


Output

It categorises kubernetes objects based on change in ApiVersion. Categories are

  1. Removed ApiVersion
  2. Deprecated ApiVersion
  3. Newer ApiVersion
  4. Unchanged ApiVersion

Within each category it identifies migration path to newer ApiVersion, possible paths are

  1. It cannot be migrated as there are no common ApiVersions between source and target kubernetes version
  2. It can be migrated but has some issues which needs to be resolved
  3. It can be migrated with just ApiVersion change

This activity is performed for both current and new ApiVersion.

Other Similar Tools

  1. kubeval - most popular, only validates against the given kubernetes version, doesn't provide migration path
  2. kube-no-trouble - provides information about removed and deprecated api but doesnt validate schema
  3. kubepug - provides information based on deprecation comments in the schema, doesn't provide information
Owner
Devtron Labs
Software Delivery Workflow For Kubernetes
Devtron Labs
Comments
  • panic: runtime error: invalid memory address or nil pointer dereference

    panic: runtime error: invalid memory address or nil pointer dereference

    When running kubedd with no arguments I get a panic:

    goroutine 1 [running]:
    github.com/devtron-labs/silver-surfer/pkg.(*Cluster).ServerVersion(0xc0001b3940)
    	/go/src/silver-surfer/pkg/Cluster.go:60 +0x18
    github.com/devtron-labs/silver-surfer/kubedd.ValidateCluster(0x0, 0xc00046e8c0)
    	/go/src/silver-surfer/kubedd/kubedd.go:95 +0x145
    main.processCluster()
    	/go/src/silver-surfer/main.go:168 +0x251
    main.glob..func1(0x2b2a240, {0x2b6f4c8, 0x0, 0x0})
    	/go/src/silver-surfer/main.go:108 +0x12c
    github.com/spf13/cobra.(*Command).execute(0x2b2a240, {0xc000198240, 0x0, 0x0})
    	/go/src/silver-surfer/vendor/github.com/spf13/cobra/command.go:766 +0x5f8
    github.com/spf13/cobra.(*Command).ExecuteC(0x2b2a240)
    	/go/src/silver-surfer/vendor/github.com/spf13/cobra/command.go:852 +0x2dc
    github.com/spf13/cobra.(*Command).Execute(...)
    	/go/src/silver-surfer/vendor/github.com/spf13/cobra/command.go:800
    main.Execute()
    	/go/src/silver-surfer/main.go:258 +0x25
    main.main()
    	/go/src/silver-surfer/main.go:286 +0x17
    
  • Color output unreadable in terminal with light background.

    Color output unreadable in terminal with light background.

    Hello,

    Unfortunately output is hardly readable in terminal with light background, is it possible don't use color at all or just leave "background" for all lines? Selection_360

    Thanks, Stan

  • fix: failing docker builds

    fix: failing docker builds

    The Problem

    As mentioned in #23, docker builds were failing with the error message

    Error response from daemon: failed to parse Dockerfile: ENV must have two arguments
    

    The cause of the issue was that the ENV command in dockerfile expects two values - the name of the environment variable, and its value. The GITHUB_TOKEN environment variable had no value set against it, causing docker builds to fail

    https://github.com/devtron-labs/silver-surfer/blob/45804592c4062d603bec137802a37003e6a4341a/Dockerfile#L9

    The Fix

    This PR resolves #23

    Added a new build argument named AUTH_TOKEN that would then be set as the value for the GITHUB_TOKEN environment variable in dockerfile

    Updated the docker build command in README!

  • failed to unmarshal extension properties: invalid character ':' after top-level value (404: Not Found)

    failed to unmarshal extension properties: invalid character ':' after top-level value (404: Not Found)

    When running it against an AWS EKS 1.19 cluster I get the following error:

    ❯ ./bin/kubedd --target-kubernetes-version 1.20
    ERR  - failed to unmarshal extension properties: invalid character ':' after top-level value (404: Not Found)
    
    Results for cluster at version 1.19+ to 1.20
    -------------------------------------------
    >>>> Deprecated API Version's <<<<
    ...
    

    The tool seems to work though, so I'm not sure what the error actually means.

    Built from source with make build from commit 086224d8d4ec6fe2ddf753bc872250e5afcd6cb5.

  • project/tool name (silver-surfer or kubedd)

    project/tool name (silver-surfer or kubedd)

    As I couldn't find an obvious or easy explanation, I'll just ask in this issue:

    What's up with the name?

    As far as I understand it, the only binary in this project is called kubedd, yet there are multiple references to silver-surfer. Is the latter just an internal name or will silver-surfer for example eventually be a collection of different tools and binaries with kubedd just being the first one?

    It is a bit confusing to see that Kubedd is the first word in the Readme and the releases are named silver-surfer.

  • Q: How to see all issues?

    Q: How to see all issues?

    How can I see which 4 issues I have to fix for the following result?

    >>>> Newer Versions available <<<<
     Namespace         Name                                         Kind           API Version (Current Available)   Replace With API Version (Latest Available)   Migration Status                                   
     my-namespace   my-ingress                                 Ingress        networking.k8s.io/v1beta1         networking.k8s.io/v1                          4 issue(s): fix issues before migration
    

    If it's too much to print detailed output in this tabular form, maybe we can have a different output format (e.g. json)?

  • False positive for API deprecation 1.16 for field last-applied-configuration even when apiVersion is correctly set

    False positive for API deprecation 1.16 for field last-applied-configuration even when apiVersion is correctly set

    The apiVersion is clearly set as apps/v1, however in the last-applied-configuration field there is a mention of extensions/v1beta1. The deployment is listed by kubedd as having the deprecation.

    Example:

    apiVersion: apps/v1
    kind: Deployment
    metadata:
      annotations:
        deployment.kubernetes.io/revision: "37"
        kubectl.kubernetes.io/last-applied-configuration: |
          {"apiVersion":"extensions/v1beta1","kind":"Deployment" ........
    
  • Issue building docker image with dockerfile

    Issue building docker image with dockerfile

    I'm running into an error while building docker image using the command

    docker build . -t silver-surfer:v1.0 --build-arg RELEASE=goreleaser --build-arg auth=YOUR_GITHUB_TOKEN
    

    The error message says

    Error response from daemon: failed to parse Dockerfile: ENV must have two arguments
    
  • build error: missing requirements

    build error: missing requirements

    Just did a fresh clone Tue Sep 21 12:42:46 2021 +0530 and getting the following build error when running the make command.

    go build -o bin/kubedd .
    kubedd/kubedd.go:23:2: package github.com/devtron-labs/silver-surfer/pkg imports github.com/mgutz/ansi from implicitly required module; to add missing requirements, run:
    	go get github.com/mgutz/[email protected]
    make: *** [build] Error 1
    

    I ran the following command to resolve the issue.

    go mod tidy
    go mod vendor
    

    After that make was able to run and the binary executes OK.

  • can we create new tag please?

    can we create new tag please?

    if we compare the latest tag and master branch .. from output of latest tag (v0.1.1) ..I see the below error

    ERR  - failed to unmarshal extension properties: invalid character ':' after top-level value (404: Not Found)
    

    In master branch I don't see this.

  • kubedd reports unsupported property for data (RawExtension)

    kubedd reports unsupported property for data (RawExtension)

    Kubedd reports many errors about not supported property (spec). This happens for ControllerRevision objects in our cluster:

     falco                audit-falco-558b87fb8c                                 ControllerRevision   apps/v1                           data    property "spec" is unsupported  
     falco                audit-falco-594bd6b8d8                                 ControllerRevision   apps/v1                           data    property "spec" is unsupported  
     falco                audit-falco-59c55fcdbd                                 ControllerRevision   apps/v1                           data    property "spec" is unsupported  
     falco                audit-falco-5df9d6bd48                                 ControllerRevision   apps/v1                           data    property "spec" is unsupported  
    

    Example instance of ControllerRevision that kubedd complained abou:

    # Please edit the object below. Lines beginning with a '#' will be ignored,
    # and an empty file will abort the edit. If an error occurs while saving this file will be
    # reopened with the relevant failures.
    #
    apiVersion: apps/v1
    data:
      spec:
        template:
          $patch: replace
          metadata:
            annotations:
              checksum/certs: 01ba4719c80b6fe911b091a2c05124b64eeece964e09c058ef8f9805daca546b
              checksum/config: c04d1feb5233533ff0f8ac8e3320213447a4d2912d69c8332f43360df11b1a18
              checksum/rules: 2c4eaf6e810932d56ee056ba47a0fe15f4eaf37081a1fe00edc62bc8ceb7a735
            creationTimestamp: null
            labels:
              app: audit-falco
              role: security
            name: audit-falco
          spec:
            containers:
            - args:
              - /usr/bin/falco
              - --cri
              - /run/containerd/containerd.sock
              - -K
              - /var/run/secrets/kubernetes.io/serviceaccount/token
              - -k
    [CUT]
    kind: ControllerRevision
    metadata:
      annotations:
        deprecated.daemonset.template.generation: "18"
        meta.helm.sh/release-name: audit-falco
        meta.helm.sh/release-namespace: falco
      creationTimestamp: "2022-01-31T14:16:41Z"
      labels:
        app: audit-falco
        controller-revision-hash: 558b872b8c
        role: security
      name: audit-falco-558b87fb8c
      namespace: falco
      ownerReferences:
      - apiVersion: apps/v1
        blockOwnerDeletion: true
        controller: true
        kind: DaemonSet
        name: audit-falco
        uid: 0a3821f7-fe77-48f4-a4ca-c7490b337ffa
      resourceVersion: "313405104"
      uid: 106621ac-e2a3-4f9b-8b6a-0352daff3fa6
    revision: 18
    
  • Update to latest

    Update to latest

    This PR does the following:

    • Removes the vendor folder as you are already instructing users to create it independently with the go mod vendor command. There is no need to include the folder here, the files will be the same as was is described in the go.sum file.

    • Updates dependencies to their latest values via go get -u ./...,

    • Updates the default target kube version to 1.24 (latest).

    • Updates the .gitingore file to include VSCode created directories/files, as well as the vendor directory.

    • Additionally, adds the k8s.io/client-go/plugin/pkg/client/auth package because when I pulled the repository initially, it still wasn't there despite a few issues and PRs that would suggest the contrary.

    • go fmt'd the whole thing which modified almost every go file...

    • applied changes as suggested by golangci-lint

  • Kubernetes update

    Kubernetes update

    Kubernetes is spelled with a capital K based on industry standards. Updated a few other sentence errors while I was at it. Hopefully this makes it more readable and user friendly.

  • Cluster Upgrade Recommendation

    Cluster Upgrade Recommendation

    Silver-Surfer crawls through the cluster and give all the possible migrations and issues, introduce the capability to generate the deployment file for the all the possible migrations of apiVersion.

kcount counts Kubernetes (K8s) objects across clusters.

kcount counts Kubernetes (K8s) objects across clusters. It gets the cluster configuration, including cluster name and namespace, from kubeconfig files

Sep 23, 2022
network-node-manager is a kubernetes controller that controls the network configuration of a node to resolve network issues of kubernetes.
network-node-manager is a kubernetes controller that controls the network configuration of a node to resolve network issues of kubernetes.

Network Node Manager network-node-manager is a kubernetes controller that controls the network configuration of a node to resolve network issues of ku

Dec 18, 2022
provider-kubernetes is a Crossplane Provider that enables deployment and management of arbitrary Kubernetes objects on clusters

provider-kubernetes provider-kubernetes is a Crossplane Provider that enables deployment and management of arbitrary Kubernetes objects on clusters ty

Dec 14, 2022
Crossplane provider to provision and manage Kubernetes objects on (remote) Kubernetes clusters.

provider-kubernetes provider-kubernetes is a Crossplane Provider that enables deployment and management of arbitrary Kubernetes objects on clusters ty

Jan 3, 2023
cluster-api-state-metrics (CASM) is a service that listens to the Kubernetes API server and generates metrics about the state of custom resource objects related of Kubernetes Cluster API.

Overview cluster-api-state-metrics (CASM) is a service that listens to the Kubernetes API server and generates metrics about the state of custom resou

Oct 27, 2022
upgrade from controller-runtime 0.6.5. Reactor substitute.

Upgrade Kubernetes Controller Runtime from v0.6.5 This blog concentrates on test case migration as the real code migration pretty detailed and straigh

Dec 6, 2021
GoMota - a mass upgrade tool for Tasmota devices.

GoMota GoMota is a mass upgrade tool for Tasmota devices. For each execution, multiple devices on the specified network can be upgraded one version, a

May 12, 2022
terraform-plugin-mux Example (upgrade(sdk/v2) + framework)

Terraform Provider Scaffolding (Terraform Plugin Framework) This template repository is built on the Terraform Plugin Framework. The template reposito

Feb 8, 2022
Hot-swap Kubernetes clusters while keeping your microservices up and running.

Okra Okra is a Kubernetes controller and a set of CRDs which provide advanced multi-cluster appilcation rollout capabilities, such as canary deploymen

Nov 23, 2022
Write controller-runtime based k8s controllers that read/write to git, not k8s

Git Backed Controller The basic idea is to write a k8s controller that runs against git and not k8s apiserver. So the controller is reading and writin

Dec 10, 2021
K8s-cinder-csi-plugin - K8s Pod Use Openstack Cinder Volume

k8s-cinder-csi-plugin K8s Pod Use Openstack Cinder Volume openstack volume list

Jul 18, 2022
K8s-go-structs - All k8s API Go structs

k8s-api go types Why? Its nice to have it all in a single package. . |-- pkg |

Jul 17, 2022
Scout for alarming issues in your Kubernetes cluster
Scout for alarming issues in your Kubernetes cluster

Kube-Scout An alerting tool for Kubernetes clusters issues of all types, in real time, with intelligent redundancy, and easily extendable api. Kube-Sc

Dec 20, 2022
Controller-check - Run checks against K8s controllers to verify if they meets certain conventions

controller-check Run checks against K8s controllers to verify if they meets cert

Jan 4, 2022
A k8s vault webhook is a Kubernetes webhook that can inject secrets into Kubernetes resources by connecting to multiple secret managers
A k8s vault webhook is a Kubernetes webhook that can inject secrets into Kubernetes resources by connecting to multiple secret managers

k8s-vault-webhook is a Kubernetes admission webhook which listen for the events related to Kubernetes resources for injecting secret directly from sec

Oct 15, 2022
Cloudflare-operator - Manage Cloudflare DNS records with Kubernetes objects

cloudflare-operator Documentation The goal of cloudflare-operator is to manage C

Nov 16, 2022
This repository is where I'm learning to write a CLI using Go, while learning Go, and experimenting with Docker containers and APIs.

CLI Project This repository contains a CLI project that I've been working on for a while. It's a simple project that I've been utilizing to learn Go,

Dec 12, 2021
a fast changelog generator sourced from PRs and Issues

chronicle A fast changelog generator that sources changes from GitHub PRs and issues, organized by labels. chronicle --since-tag v0.16.0 chronicle --s

Nov 19, 2022
Gh-i - Search your github issues interactively
Gh-i - Search your github issues interactively

search your github issues interactively Installation • Usage • Feedback Search G

Dec 29, 2022