operator to install cluster manager and klusterlet.

registration-operator

Minimum cluster registration and work

Community, discussion, contribution, and support

Check the CONTRIBUTING Doc for how to contribute to the repo.

How to Deploy

Deploy all-in-one deployment on kind

  1. Create a kind cluster 
    kind create cluster --name cluster1
kind get kubeconfig --name cluster1 > ./.kubeconfig
  2. Deploy all components on the kind cluster 
    export MANAGED_CLUSTER=cluster1
make deploy
  3. To clean the environment, run make clean-deploy

Deploy on OCP

  1. Deploy hub component 
    make deploy-hub
  2. Deploy agent component 
    export KLUSTERLET_KUBECONFIG_CONTEXT={kube config context of managed cluster}
make deploy-spoke
  3. To clean the environment, run make clean-hub and make clean-spoke

What is next

After a successful deployment, a certificatesigningrequest and a managedcluster will be created on the hub.

kubectl get csr
kubectl get managedcluster

Next approve the csr and set managecluster to be accepcted by hub with the following command

kubectl certificate approve {csr name}
kubectl patch managedcluster {cluster name} -p='{"spec":{"hubAcceptsClient":true}}' --type=merge

Security Response

If you've found a security issue that you'd like to disclose confidentially please contact Red Hat's Product Security team. Details at https://access.redhat.com/security/team/contact

License

This code is released under the Apache 2.0 license. See the file LICENSE for more information.

Owner
Open Cluster Management
Open Cluster Management
https://github.com/open-cluster-management-io/registration-operator
Comments
  • Add management workload annotations

    Add management workload annotations

    In support of the workload partitioning feature (openshift/enhancements#703), we need to add annotations to all management pods and namespaces so they can be properly identified and assigned to segregated management cores on clusters configured to do so.

    Signed-off-by: Ian Miller [email protected]

  • support hubRegistrationFeatureGates and spokeRegistrationFeatureGates

    support hubRegistrationFeatureGates and spokeRegistrationFeatureGates

    This PR is depend on the PR in api https://github.com/open-cluster-management-io/api/pull/149, must modifying go.mod and updating vendor afeter api pr 149 is merged

    reconcile hubRegistrationFeatureGates and spokeRegistrationFeatureGates to registration deployment()

    Signed-off-by: ivan-cai [email protected]

  • support deploy klusterlet outside of managed cluster

    support deploy klusterlet outside of managed cluster

    Related PRs:

    • https://github.com/open-cluster-management-io/api/pull/114
    • https://github.com/open-cluster-management-io/registration/pull/175
    • https://github.com/open-cluster-management-io/work/pull/104

    related issue: open-cluster-management-io/registration-operator#158

  • Delete klusterlet namespace after klusterlet cr is removed

    Delete klusterlet namespace after klusterlet cr is removed

    This is a workaround for cluster detach. See https://github.com/open-cluster-management/backlog/issues/4565 for more details.

    Note: This change CANNOT be merged until delete permission is added for clusterrole klusterlet in rcm-controller.

  • Add controller to sync image pull secret into addon namespaces.

    Add controller to sync image pull secret into addon namespaces.

    Signed-off-by: xuezhaojun [email protected]

    Epic: https://github.com/stolostron/backlog/issues/23093

    User-story: https://github.com/stolostron/backlog/issues/23929

  • SA token ensure func did not find the right tokens

    SA token ensure func did not find the right tokens

    When the service-account name is long enough, the sa token ensure func may mismatch the token.

    For example, token names of the following service account do not have "token" characters.

    apiVersion: v1
imagePullSecrets:
- name: open-cluster-management-image-pull-credentials
- name: klusterlet-hypershift-demo-managed-clu-f203af0b-dockercfg-kgh65
kind: ServiceAccount
metadata:
  creationTimestamp: "2022-01-22T07:56:39Z"
  name: klusterlet-hypershift-demo-managed-cluster-registration-sa
  namespace: klusterlet-hypershift-demo-managed-cluster
  resourceVersion: "70692537"
  uid: 975eb940-6dbd-4ad7-a2d8-77cfd867887f
secrets:
- name: klusterlet-hypershift-demo-managed-cluster-registration-sax7zps
- name: klusterlet-hypershift-demo-managed-clu-f203af0b-dockercfg-kgh65

    cc @xuezhaojun

  • fix issue #150

    fix issue #150

    Signed-off-by: kim-fitness [email protected]

    This PR aims to enhance the operator to give the klusterlet an accurate condition. It is related with open-cluster-management-io/clusteradm#65

  • refactor degraded condition logic

    refactor degraded condition logic

    refactor degraded condition logic to make the code more clear and fix below problems:

    1. after the bootstrap, the registration agent keeps working, but the bootstrap secret will be expired, the registration degraded condition will prompt BootstrapSecretUnauthorized
    2. if the hub cluster-admin approve the csr, but does not accept the managed cluster, at this time point, the registration and work agent degraded conditions are not consistent, registration agent degraded condition is HubConfigSecretUnauthorized, but work agent degraded condition is KubeConfigMissing
  • Fix the paradox description of the klusterlet condition

    Fix the paradox description of the klusterlet condition

    Signed-off-by: zhujian [email protected]

    Fix https://github.com/open-cluster-management-io/registration-operator/issues/289

    /assign @qiujian16 @zhiweiyin318

  • Make work webhook feature gate configurable

    Make work webhook feature gate configurable

    Signed-off-by: zhujian [email protected]

    issue reference: https://github.com/open-cluster-management-io/work/issues/156

    user can use the following config for the clustermanager to enable or disable the features for the work webhook

    spec:
  workConfiguration:
    featureGates:
    - feature: NilExecutorValidating
      mode: Enable
  • Add HubApiServerHostAlias for registration-agent and work-agent

    Add HubApiServerHostAlias for registration-agent and work-agent

    Update dependency of open-cluster-management.io/api , and fix a typo.

    Add hostAliases rendering logic into registration-agent and work-agent's YAML template files.

    Signed-off-by: Bowen Zhu [email protected]

  • Show progress of ClusterManager terminating with conditions in the status

    Show progress of ClusterManager terminating with conditions in the status

    Add new conditions to the status of ClusterManager to show the current progress of terminating. It helps users to find out where the terminating process is stuck.

  • Missing addonplacementscores in cluster role

    Missing addonplacementscores in cluster role

    K8S Version: 1.21.2 OLM Version: 0.20.0

    When cluster manager in installed through OLM the clusterrole open-cluster-management:cluster-manager-placement:controller has not the addonplacementscores permission causing pod cluster-manager-placement-controllerto generate this error:

    k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1alpha1.AddOnPlacementScore: failed to list *v1alpha1.AddOnPlacementScore: addonplacementscores.cluster.open-cluster-management.io is forbidden: User "system:serviceaccount:open-cluster-management-hub:cluster-manager-placement-controller-sa" cannot list resource "addonplacementscores" in API group "cluster.open-cluster-management.io" at the cluster scope

    If the operator is installed through CLI the clusterrole contains the addonplacementscores permission

    Proposal:

    Add this commit into cluster role file for olm. Then you need to release a minor version 0.6.1

  • The

    The "MANAGED CLUSTER URLS" displayed is incorrect by registered cluster

    The registered cluster url is displayed as "https://localhost". Look at the following example.

    kubectl get managedcluster

    NAME HUB ACCEPTED MANAGED CLUSTER URLS JOINED AVAILABLE AGE cluster1 true https://localhost True True 5h35m

    I think the newly registered cluster ip should be displayed here as shown below. https://clusterip:6443

    I will also investigate this issue.

  • klusterlet-work-agent instability

    klusterlet-work-agent instability

    Commands used to depoly a hub and managed cluster using minikube:

    kubectl apply -f https://raw.githubusercontent.com/kubernetes/cluster-registry/master/cluster-registry-crd.yaml
git clone https://github.com/open-cluster-management/registration-operator.git
cd ./registration-operator
git checkout master
make deploy-hub
kubectl wait deployments --all --for condition=Available -n olm --timeout 1m
kubectl wait deployments --all --for condition=Available -n open-cluster-management
kubectl wait deployments --all --for condition=Available -n open-cluster-management-hub --timeout 2m
minikube start --driver=kvm2 --profile=cluster1
kubectl config view --flatten --context=hub --minify >/tmp/hub-config
kubectl config view --flatten --context=cluster1 --minify >/tmp/cluster1-config
kubectl config use-context cluster1
export KLUSTERLET_KIND_KUBECONFIG=/tmp/cluster1-config
export KIND_CLUSTER=cluster1
export HUB_KIND_KUBECONFIG=/tmp/hub-config
make deploy-spoke-kind
kubectl wait deployments --all --for condition=Available -n open-cluster-management
kubectl wait deployments --all --for condition=Available -n open-cluster-management-agent

    Command to gather state:

    NAME                                             READY   STATUS             RESTARTS   AGE
klusterlet-registration-agent-85f68b68c4-hr7ln   1/1     Running            0          32m
klusterlet-registration-agent-85f68b68c4-pkg98   1/1     Running            0          32m
klusterlet-registration-agent-85f68b68c4-t756z   1/1     Running            0          32m
klusterlet-work-agent-786b47fb99-7cp4x           0/1     CrashLoopBackOff   7          32m
klusterlet-work-agent-786b47fb99-8wcdb           0/1     CrashLoopBackOff   7          32m
klusterlet-work-agent-786b47fb99-vx2qk           0/1     CrashLoopBackOff   7          32m

$ minikube profile list
|----------|-----------|---------|----------------|------|---------|---------|-------|
| Profile  | VM Driver | Runtime |       IP       | Port | Version | Status  | Nodes |
|----------|-----------|---------|----------------|------|---------|---------|-------|
| cluster1 | kvm2      | docker  | 192.168.39.92  | 8443 | v1.20.2 | Running |     1 |
| hub      | kvm2      | docker  | 192.168.39.156 | 8443 | v1.20.2 | Running |     1 |
|----------|-----------|---------|----------------|------|---------|---------|-------|

$ kubectl describe pod klusterlet-work-agent-786b47fb99-7cp4x -n open-cluster-management-agent
Name:         klusterlet-work-agent-786b47fb99-7cp4x
Namespace:    open-cluster-management-agent
Priority:     0
Node:         cluster1/192.168.39.92
Start Time:   Tue, 13 Apr 2021 08:16:03 -0700
Labels:       app=klusterlet-manifestwork-agent
              pod-template-hash=786b47fb99
Annotations:  <none>
Status:       Running
IP:           172.17.0.16
IPs:
  IP:           172.17.0.16
Controlled By:  ReplicaSet/klusterlet-work-agent-786b47fb99
Containers:
  klusterlet-manifestwork-agent:
    Container ID:  docker://0247a6d965fc6040fab94d2cacf7bfbea915ac92b3b7cb54ad1e2c54da05ceb9
    Image:         quay.io/open-cluster-management/work
    Image ID:      docker-pullable://quay.io/open-cluster-management/work@sha256:e60ddb4fdcb24932816118f564748126ff032f49354f4d430d2d46a9df22a3f5
    Port:          <none>
    Host Port:     <none>
    Args:
      /work
      agent
      --spoke-cluster-name=cluster1
      --hub-kubeconfig=/spoke/hub-kubeconfig/kubeconfig
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Tue, 13 Apr 2021 08:46:26 -0700
      Finished:     Tue, 13 Apr 2021 08:47:36 -0700
    Ready:          False
    Restart Count:  7
    Requests:
      cpu:        100m
      memory:     128Mi
    Liveness:     http-get https://:8443/healthz delay=2s timeout=1s period=10s #success=1 #failure=3
    Readiness:    http-get https://:8443/healthz delay=2s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /spoke/hub-kubeconfig from hub-kubeconfig-secret (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from klusterlet-work-sa-token-ls59n (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  hub-kubeconfig-secret:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  hub-kubeconfig-secret
    Optional:    false
  klusterlet-work-sa-token-ls59n:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  klusterlet-work-sa-token-ls59n
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  40m                   default-scheduler  Successfully assigned open-cluster-management-agent/klusterlet-work-agent-786b47fb99-7cp4x to cluster1
  Normal   Pulled     39m                   kubelet            Successfully pulled image "quay.io/open-cluster-management/work" in 29.427044917s
  Normal   Pulled     38m                   kubelet            Successfully pulled image "quay.io/open-cluster-management/work" in 1.508110447s
  Normal   Pulled     37m                   kubelet            Successfully pulled image "quay.io/open-cluster-management/work" in 1.442129606s
  Normal   Pulled     33m                   kubelet            Successfully pulled image "quay.io/open-cluster-management/work" in 1.516616046s
  Normal   Started    33m (x4 over 39m)     kubelet            Started container klusterlet-manifestwork-agent
  Warning  Unhealthy  25m                   kubelet            Readiness probe failed: Get "https://172.17.0.16:8443/healthz": dial tcp 172.17.0.16:8443: connect: connection refused
  Normal   Pulled     25m                   kubelet            Successfully pulled image "quay.io/open-cluster-management/work" in 1.41603949s
  Normal   Created    25m (x5 over 39m)     kubelet            Created container klusterlet-manifestwork-agent
  Normal   Pulling    10m (x8 over 40m)     kubelet            Pulling image "quay.io/open-cluster-management/work"
  Warning  BackOff    5m17s (x72 over 37m)  kubelet            Back-off restarting failed container


$ kubectl logs -p klusterlet-work-agent-786b47fb99-7cp4x -n open-cluster-management-agent
W0413 15:46:26.585511       1 cmd.go:204] Using insecure, self-signed certificates
I0413 15:46:27.236528       1 observer_polling.go:159] Starting file observer
I0413 15:46:27.309689       1 builder.go:240] work-agent version v2.3.0-2021-03-29-10-25-04-2-g4832b07-4832b07
W0413 15:46:27.803860       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256' detected.
W0413 15:46:27.803995       1 secure_serving.go:69] Use of insecure cipher 'TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256' detected.
I0413 15:46:27.807585       1 leaderelection.go:243] attempting to acquire leader lease open-cluster-management-agent/work-agent-lock...
I0413 15:46:27.815226       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
0413 15:46:27.816318       1 shared_informer.go:240] Waiting for caches to sync for RequestHeaderAuthRequestController
I0413 15:46:27.816743       1 secure_serving.go:197] Serving securely on [::]:8443
I0413 15:46:27.817189       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0413 15:46:27.815911       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0413 15:46:27.817987       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0413 15:46:27.815926       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0413 15:46:27.819358       1 shared_informer.go:240] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0413 15:46:27.816294       1 dynamic_serving_content.go:130] Starting serving-cert::/tmp/serving-cert-328065387/tls.crt::/tmp/serving-cert-328065387/tls.key
I0413 15:46:27.917412       1 shared_informer.go:247] Caches are synced for RequestHeaderAuthRequestController
I0413 15:46:27.920012       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0413 15:46:27.920228       1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0413 15:47:36.163633       1 leaderelection.go:253] successfully acquired lease open-cluster-management-agent/work-agent-lock
I0413 15:47:36.169819       1 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"open-cluster-management-agent", Name:"work-agent-lock", UID:"972836a0-5311-4022-8f44-09e7170d5a61", APIVersion:"v1", ResourceVersion:"3585", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' klusterlet-work-agent-786b47fb99-7cp4x_ce3f539c-5804-4d88-997f-7ba502de845d became leader
W0413 15:47:36.188407       1 builder.go:99] graceful termination failed, controllers failed with error: invalid configuration: [unable to read client-cert /spoke/hub-kubeconfig/tls.crt for default-auth due to open /spoke/hub-kubeconfig/tls.crt: no such file or directory, unable to read client-key /spoke/hub-kubeconfig/tls.key for default-auth due to open /spoke/hub-kubeconfig/tls.key: no such file or directory]


$ go version
go version go1.16.2 linux/amd64
$ kustomize version
{Version:kustomize/v4.0.5 GitCommit:9e8e7a7fe99ec9fbf801463e8607928322fc5245 BuildDate:2021-03-08T20:53:03Z GoOs:linux GoArch:amd64}
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.0", GitCommit:"cb303e613a121a29364f75cc67d3d580833a7479", GitTreeState:"clean", BuildDate:"2021-04-08T16:31:21Z", GoVersion:"go1.16.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}
$ minikube version
minikube version: v1.18.1
commit: 09ee84d530de4a92f00f1c5dbc34cead092b95bc

    I wonder if the last line of the logs explains why a terminaton failed, or why a termination occurred. I'm willing to run more commands to debug. This issue may be reproduced easily.

    master branch was used. head commit identifier: daf5b5cf51a7da18243c6f99cee94f8197f53b90

  • failed to deploy on kind

    failed to deploy on kind

    What happened: When trying to deploy as per all-in-one. An error raised:

    INFO[0005] Waiting for ClusterServiceVersion "open-cluster-management/klusterlet.v0.2.0" to reach 'Succeeded' phase 
INFO[0005]   Waiting for ClusterServiceVersion "open-cluster-management/klusterlet.v0.2.0" to appear 
INFO[0010]   Found ClusterServiceVersion "open-cluster-management/klusterlet.v0.2.0" phase: Pending
FATA[0600] Failed to run operator using OLM: error waiting for CSV to install: timed out waiting for the condition

    How to reproduce it (as minimally and precisely as possible):

    # kind create cluster --name hub --image=kindest/node:v1.18.0
# git clone https://github.com/RainbowMango/registration-operator.git
# cd registration-operator/
# export KIND_CLUSTER=hub
# make deploy

    cc @qiujian16

Related tags
DevOps Tools registration-operator
Operator Permissions Advisor is a CLI tool that will take a catalog image and statically parse it to determine what permissions an Operator will request of OLM during an install

Operator Permissions Advisor is a CLI tool that will take a catalog image and statically parse it to determine what permissions an Operator will request of OLM during an install. The permissions are aggregated from the following sources:

Apr 22, 2022
Basic Kubernetes operator that have multiple versions in CRD. This operator can be used to experiment and understand Operator/CRD behaviors.

add-operator Basic Kubernetes operator that have multiple versions in CRD. This operator can be used to experiment and understand Operator/CRD behavio

Dec 15, 2021
An operator which complements grafana-operator for custom features which are not feasible to be merged into core operator

Grafana Complementary Operator A grafana which complements grafana-operator for custom features which are not feasible to be merged into core operator

Aug 16, 2022
Stop using install.sh! Start using install.yml! DAJE...

Daje - A general purpose Ansible dotfiles installer Configure one time and run everywhere. What is this I've never liked using a big bash script to in

Nov 14, 2022
Kubernetes Operator Samples using Go, the Operator SDK and OLM
 Kubernetes Operator Samples using Go, the Operator SDK and OLM

Kubernetes Operator Patterns and Best Practises This project contains Kubernetes operator samples that demonstrate best practices how to develop opera

Nov 24, 2022
The missing package manager for golang binaries (its homebrew for "go install")

Bingo: The missing package manager for golang binaries (its homebrew for "go install") Do you love the simplicity of being able to download & compile

Oct 31, 2022
Kubegres is a Kubernetes operator allowing to create a cluster of PostgreSql instances and manage databases replication, failover and backup.

Kubegres is a Kubernetes operator allowing to deploy a cluster of PostgreSql pods with data replication enabled out-of-the box. It brings simplicity w

Dec 30, 2022
The Elastalert Operator is an implementation of a Kubernetes Operator, to easily integrate elastalert with gitops.

Elastalert Operator for Kubernetes The Elastalert Operator is an implementation of a Kubernetes Operator. Getting started Firstly, learn How to use el

Jun 28, 2022
Test Operator using operator-sdk 1.15

test-operator Test Operator using operator-sdk 1.15 operator-sdk init --domain rbt.com --repo github.com/ravitri/test-operator Writing kustomize manif

Dec 28, 2021
Minecraft-operator - A Kubernetes operator for Minecraft Java Edition servers

Minecraft Operator A Kubernetes operator for dedicated servers of the video game

Dec 15, 2022
K8s-network-config-operator - Kubernetes network config operator to push network config to switches

Kubernetes Network operator Will add more to the readme later :D Operations The

May 16, 2022
a k8s operator 、operator-sdk

helloworld-operator a k8s operator 、operator-sdk Operator 参考 https://jicki.cn/kubernetes-operator/ https://learnku.com/articles/60683 https://opensour

Jan 27, 2022
Terraform-operator - The Terraform Operator provides support to run Terraform modules in Kubernetes in a declaritive way as a Kubernetes manifest

Terraform Operator The Terraform Operator provides support to run Terraform modu

Oct 19, 2022
Pulumi-k8s-operator-example - OpenGitOps Compliant Pulumi Kubernetes Operator Example

Pulumi GitOps Example OpenGitOps Compliant Pulumi Kubernetes Operator Example Pr

May 6, 2022
Nebula Operator manages NebulaGraph clusters on Kubernetes and automates tasks related to operating a NebulaGraph cluster

Nebula Operator manages NebulaGraph clusters on Kubernetes and automates tasks related to operating a NebulaGraph cluster. It evolved from NebulaGraph Cloud Service, makes NebulaGraph a truly cloud-native database.

Dec 31, 2022
PolarDB-X Operator is a Kubernetes extension that aims to create and manage PolarDB-X cluster on Kubernetes.

GalaxyKube -- PolarDB-X Operator PolarDB-X Operator is a Kubernetes extension that aims to create and manage PolarDB-X cluster on Kubernetes. It follo

Dec 19, 2022
YurtCluster Operator creates and manages OpenYurt cluster atop Kubernetes

YurtCluster Operator Quick Start Prepare a Kubernetes cluster # cat <<EOF | kind create cluster --config=- kind: Cluster apiVersion: kind.x-k8s.io/v1a

Aug 3, 2022
Kubernetes Operator for MySQL NDB Cluster.

MySQL NDB Operator The MySQL NDB Operator is a Kubernetes operator for managing a MySQL NDB Cluster setup inside a Kubernetes Cluster. This is in prev

Dec 21, 2022
Sbom-operator - Catalogue all images of a Kubernetes cluster to multiple targets with Syft

sbom-operator Catalogue all images of a Kubernetes cluster to multiple targets w

Jan 4, 2023