Add a new parseHelmChart native function that expands helm charts into jsonnet objects. The new code links helm (v3) libraries directly, and does not require nor use the external helm command.

Note the helm chart is provided as an array of bytes (numbers 0-255) - this builtin does not fetch the helm chart. The expectation is that this will be used with another mechanism like the jsonnet importbin statement.

Fixes #267

Example usage:

local kubecfg = import "kubecfg.libsonnet";

local data = importbin "https://charts.jetstack.io/charts/cert-manager-v1.5.3.tgz";
local cm = kubecfg.parseHelmChart(data, "cert-manager", namespace, {
  // Example values.yaml
  webhook: {replicaCount: 2},
});

// ... returns a jsonnet object with filename keys from the helm chart
// and expanded/parsed Kubernetes objects as values.

// The result can be manipulated and used just like any other jsonnet
// value.
cm + {
  "cert-manager/templates/webhook-deployment.yaml": [o + {
    spec+: {
      template+: {
        spec+: {
          nodeSelector+: {"kubernetes.io/arch": "amd64"},
        },
      },
    },
  } for o in super["cert-manager/templates/webhook-deployment.yaml"]],
}

Caveats:

• Helm 'hooks' are not supported and ignored.
• Chart sort order is ignored, and the usual kubecfg sort mechanism is used.
• Probably some other things.

Add some helper functions to help manipulate jsonnet structures that conform to kubecfg's usual "nested collections of Kubernetes objects" result structure.

Specifically they handle:

• null (ignored)
• single Kubernetes object
• Kubernetes v1.List of Kubernetes objects (note other kind-specific list types are not supported)
• jsonnet array of any of these types (recursively)
• jsonnet object with values of any of these types (recursively)

Helper functions added in this commit:

isK8sObject: returns true if this is a Kubernetes object.

deepMap: map Kubernetes objects, preserving the original nested structure.

fold: iterate over Kubernetes objects and accumulate a result.

gvkName: useful function for fold. Accumulates a two-level nested object by apiVersion.kind, then object name.

gvkNsName: useful function for fold. Accumulates a three-level nested object by apiVersion.kind, object namespace (or _), then object name.

This addresses https://github.com/kubecfg/kubecfg/issues/52

Adds a new cli flag --gc-all-namespaces to kubecfg update.

• When set to false: apply the namespace scope (i.e. default namespace, adjustable with --namespace) to garbage collection
• When set to true (default): keep behavior unchanged

This is an import of https://github.com/anguslees/kubecfg-1/tree/helm onto the new kubecfg repository

Add a new helmTemplate native function that downloads and expands helm charts into jsonnet objects. The new code links helm (v3) libraries directly, and does not require nor use the external helm command.

Fixes vmware-archive#267

Example usage:

local kubecfg = import "kubecfg.libsonnet";

local url = "https://charts.jetstack.io/charts/cert-manager-v1.5.3.tgz";
local cm = kubecfg.helmTemplate("cert-manager", namespace, url, {
  // Example values.yaml
  webhook: {replicaCount: 2},
});

// ... returns a jsonnet object with filename keys from the helm chart
// and expanded/parsed Kubernetes objects as values.

// The result can be manipulated and used just like any other jsonnet
// value.
cm + {
  "cert-manager/templates/webhook-deployment.yaml"+: {
    spec+: {
      template+: {
        spec+: {
          nodeSelector+: {"kubernetes.io/arch": "amd64"},
        },
      },
    },
  },
}

Caveats:

• Helm 'hooks' are not supported and ignored.
• Chart sort order is ignored, and the usual kubecfg sort mechanism is used.
• chartURL argument uses the kubecfg URL-based importer, but will reject relative URLs by default[1].
• HTTP_PROXY is obeyed, but there is no other cache. The helm chart is re-downloaded on every invocation (for now).
• Probably some other things.

[1]: Relative URLs can be enabled using a new --allow-relative-helm-urls flag. URLs are interpreted relative to $PWD currently, even when used by jsonnet from remote URLs. This will change. TODO: Make this consistent with usual relative import semantics, and enable by default.

In our internal project, we have almost 5000 individual YAML files that we generate. For historic reasons, we continue to name the files .yml instead of .yaml.

By enabling kubecfg to write the files with a .yml suffix, we can produce these files for our CI/CD pipeline measurably faster.

I used hyperfine to benchmark the current mode (my build of kubecfg, do not specify --format, followed by find | read | mv), vs. the same build of kubecfg with --format=yml, no call to find.

Benchmark #1: GENMODE=current make generate Time (mean ± σ): 13.384 s ± 0.459 s [User: 79.086 s, System: 19.813 s] Range (min … max): 12.775 s … 14.280 s 10 runs

Benchmark #2: GENMODE=newkubecfg make generate Time (mean ± σ): 11.166 s ± 0.216 s [User: 73.907 s, System: 9.905 s] Range (min … max): 10.858 s … 11.544 s 10 runs

Summary 'GENMODE=newkubecfg make generate' ran 1.20 ± 0.05 times faster than 'GENMODE=current make generate'

This PR contains the following updates:

| Package | Type | Update | Change | |---|---|---|---| | github.com/mkmik/yaml | replace | major | v0.0.0-20210505221935-5a0cbc1c4094 -> v2.4.0 |

Release Notes

Configuration

📅 Schedule: At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

• [ ] If you want to rebase/retry this PR, click this checkbox.

This PR has been generated by WhiteSource Renovate. View repository job log here.

This PR contains the following updates:

| Package | Type | Update | Change | |---|---|---|---| | github.com/googleapis/gnostic | require | minor | v0.5.5 -> v0.6.2 |

Release Notes

Configuration

📅 Schedule: At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

• [ ] If you want to rebase/retry this PR, click this checkbox.

This PR has been generated by WhiteSource Renovate. View repository job log here.

kubecfg supports garbage collection with a tag given by --gc-tag. This PR adds another flag --gc-tags-from-input which instructs kubecfg to garbage collect for all tags present on objects in the input.

This allows to deploy multiple applications at once where each application has its own gc tag.

Demo:

$ export KUBECFG_ALPHA=true # or pass the --alpha flag
$ ./kubecfg eval ./examples/guestbook.jsonnet  | head
frontend:
  deploy:
    apiVersion: apps/v1beta2
    kind: Deployment
    metadata:
      annotations: {}
      labels:
        name: frontend
      name: frontend
    spec:
$ ./kubecfg eval ./examples/guestbook.jsonnet  -k    
- frontend
- master
- slave

$ ./kubecfg eval ./examples/guestbook.jsonnet -e $.master -k
- deploy
- svc

$ ./kubecfg eval ./examples/guestbook.jsonnet -e $.master.svc
apiVersion: v1
kind: Service
metadata:
  annotations: {}
  labels:
    name: redis-master
  name: redis-master
spec:
  ports:
  - port: 6379
    targetPort: 6379
  selector:
    name: redis-master
  type: ClusterIP

Might as well include this, since it's supported. I wanted to add fish completion to the AUR pkg for kubecfg, so this is a necessary first step.

NOTES:

1. I tried to run make generate as advised by the contrib guide, but it doesn't seem to exist. Not sure what's going on there. Maybe those need an update?
2. I ran make tidy as a test, and it did produce various changes, notably

diff --git a/go.mod b/go.mod
index 193c90c..f1c7448 100644
--- a/go.mod
+++ b/go.mod
@@ -31,7 +31,6 @@ require (
 )

 require (
-       cloud.google.com/go v0.100.2 // indirect

but I decided not to include this in my PR, insofar as this seems like a pretty trivial change to me. Please advise if this is wrong.

3. I tested ./kubecfg completion > ~/.config/fish/completions/kubecfg.fish on fish 3.5.1 and it seems to work as expected, but I didn't go deeper than that.

To reproduce the issue:

local i = {
  kind: 'Ingress',
  metadata: { name: 'i1', namespace: 'ns1' },
  spec: {},
};

[
  (i { apiVersion: 'networking.k8s.io/v1' }),
  (i { apiVersion: 'networking.k8s.io/v1' }),
]

This fails. The following should also fail, but works as expected:

local i = {
  kind: 'Ingress',
  metadata: { name: 'i1', namespace: 'ns1' },
  spec: {},
};

[
  (i { apiVersion: 'networking.k8s.io/v1' }),
  (i { apiVersion: 'networking.k8s.io/v1beta1' }),
]

Kubernetes would treat `networking.k8s.io/v1` and `networking.k8s.io/v1beta1` resources with same name and namespace as same, so it is a duplicate.

I'm not sure how exactly should `apiVersion` be compared.

This PR contains the following updates:

| Package | Type | Update | Change | |---|---|---|---| | actions/cache | action | digest | 9b0c1fc -> 4723a57 |

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

• [ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate. View repository job log here.

This PR contains the following updates:

| Package | Type | Update | Change | |---|---|---|---| | github.com/onsi/ginkgo | require | major | v1.16.5 -> v2.6.1 |

Release Notes

It("is interruptible", func(ctx SpecContext) { // or context.Context instead of SpecContext, both are valid.
    // do things until `ctx.Done()` is closed, for example:
    req, err := http.NewRequestWithContext(ctx, "POST", "/build-widgets", nil)
    Expect(err).NotTo(HaveOccured())
    _, err := http.DefaultClient.Do(req)
    Expect(err).NotTo(HaveOccured())

    Eventually(client.WidgetCount).WithContext(ctx).Should(Equal(17))
}, NodeTimeout(time.Second*20), GracePeriod(5*time.Second))

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

• [ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate. View repository job log here.

This PR contains the following updates:

| Package | Type | Update | Change | |---|---|---|---| | actions/checkout | action | digest | 93ea575 -> 755da8c |

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

• [ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate. View repository job log here.

2022/12/20 16:57:28 Using base distroless.dev/static:latest@sha256:dd37b804e4de19f7d979c719e3520a79651d0158cafc27fe2005d099a0029afd for github.com/kubecfg/kubecfg
2022/12/20 16:57:28 Building github.com/kubecfg/kubecfg for linux/amd64
2022/12/20 16:57:28 Building github.com/kubecfg/kubecfg for linux/ppc64le
2022/12/20 16:57:28 Building github.com/kubecfg/kubecfg for linux/s390x
2022/12/20 16:57:28 Building github.com/kubecfg/kubecfg for linux/386
2022/12/20 16:57:28 Building github.com/kubecfg/kubecfg for linux/riscv64
2022/12/20 16:57:28 Building github.com/kubecfg/kubecfg for linux/arm64
2022/12/20 16:57:28 Building github.com/kubecfg/kubecfg for linux/arm/v7
2022/12/20 16:57:29 Building github.com/kubecfg/kubecfg for linux/arm/v6

Initially we implement gc-tags in annotations but that makes it slow for us to find objects matching the tag (it requires a full scan of the apiserver), so we're migrating to using labels.

However @muxmuse found out that:

Some automatically created resources inherit labels of their "parents", but not annotations. E.g. Endpoints inherit labels (but not annotations) of the related Service and don't get an ownerReference, such that they will be garbage-collected on each kubecfg update when using tag-based garbage collection.

Reproducible on docker-desktop v1.25.0 with

kubecfg update --gc-tag=test main.yaml
kubecfg update --gc-tag=test main.yaml
# INFO  Garbage collecting endpoints default.main (v1)
# main.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: main
spec:
  selector:
    matchLabels:
      app: main
  template:
    metadata:
      labels:
        app: main
    spec:
      containers:
      - name: main
        image: containous/whoami
---
apiVersion: v1
kind: Service
metadata:
  name: main
spec:
  ports:
  - port: 80
  selector:
    app: main
  type: ClusterIP

This also applies to metrics.k8s.io/PodMetrics which inherit labels of the Pods they belong to, but get no ownerReference.

This PR contains the following updates:

| Package | Type | Update | Change | |---|---|---|---| | golang.org/x/crypto | require | minor | v0.3.0 -> v0.4.0 |

Release Notes

Configuration

📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

♻ Rebasing: Whenever PR is behind base branch, or you tick the rebase/retry checkbox.

🔕 Ignore: Close this PR and you won't be reminded about this update again.

• [ ] If you want to rebase/retry this PR, check this box

This PR has been generated by Mend Renovate. View repository job log here.