In some cases, our hub cluster's kubernetes version is less than 1.19, and its certificates.k8s.io is v1beta1. So we need to enable the feature gate V1beta1CSRAPICompatibility. Otherwise, the registration-controller will get the following error message.

W0630 07:07:55.361239       1 reflector.go:324] k8s.io/[email protected]/tools/cache/reflector.go:167: failed to list *v1.CertificateSigningRequest: the server could not find the requested resource
E0630 07:07:55.361260       1 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1.CertificateSigningRequest: failed to list *v1.CertificateSigningRequest: the server could not find the requested resource

Signed-off-by: Promacanthus [email protected]

As @qiujian16 suggested, it is split into two PRs to support addon registration. This is the first one.

The related issue: https://github.com/open-cluster-management/backlog/issues/10714

Signed-off-by: Yang Le [email protected]

add a controller to set default cluster set label for managed clusters which not set cluster set label. the label will be cluster.open-cluster-management.io/clusterset:default

Signed-off-by: ycyaoxdu [email protected]

Signed-off-by: suigh [email protected]

This change is used to report the GPU information to hub cluster, the status in the managedcluster will be updated. We will use this GPU information to get placement decision in the future.

nvidia.com/gpu is used here for GPU information because: 1: it is used by nvidia and k8s 2: if we support other GPUs (like AMD gpu) later, we needn't change the current name that only for nvidia gpu only, no compatibility issue.

The output of kc describe managedclusters is updated as following, nvidia.com/gpu is displayed:

Name:         cluster2
 ......
Spec:
  Hub Accepts Client:      true
  Lease Duration Seconds:  60
  ......
Status:
  Allocatable:
    Cpu:             64
    Memory:          256168Mi
    nvidia.com/gpu:  1
  Capacity:
    Cpu:             64
    Memory:          256468Mi
    nvidia.com/gpu:  1
  Conditions:

BTW: Thanks very much for Jian (@zhujian7), without his help, I cannot build the test env at all.

Signed-off-by: JiaHaoWei [email protected] Implement the Taints mentioned in #https://github.com/open-cluster-management-io/community/issues/48, and add ut and integration tests

This is a WIP PR. But since this could generate lots of discussions, wanted raise this and capture all the discussion through this PR.

Goal: We aim to capture a metric about the Spoke cluster that joins the Hub in Prometheus exposition format as shown

//cluster_id = OCP ID of the Cluster (need to resolve for eks, etc)
//type = K8s Distribution, e.g. OCP, EKS, etc
//version = Distribution version
//cluster_infrastructure_provider = value "Type" from cluster_infrastructure_provider
//hub_id = cluster_id of hub server
//cluster_name =User Display Name of Cluster (defaults to Id if not provided)
var managedClusterMetric = prometheus.NewGaugeVec(prometheus.GaugeOpts{
	Name: "a_managed_cluster",
	Help: "Managed Cluster being managed by ACM Hub.",
}, []string{"cluster_name", "cluster_id", "type", "version", "cluster_infrastructure_provider", "hub_id"})

Highlights:

1. Metrics endpoint is created such that it can be scraped by Prometheus. For this reason we have exposed a new API endpoint in https://github.com/bjoydeep/registration/tree/master/pkg/hub/custommetrics. We also have been told by @deads2k and @qiujian16 that we should NOT expose this endpoint - but use the default 8443 port which is available for this and already exposes some other key metrics about the performance of the controller. Moving to 8443 will be WIP.
2. For the code to work the way it is, we need to add an environment variable (ie we have fenced this off): https://github.com/bjoydeep/registration/blob/master/pkg/hub/manager.go#L88-#L102 . So unless the env variable is specifically added to the deployment yaml, this code will not get called.
3. We are working on adding e2e tests.
4. As of now, we can only gather the cluster-name of the spoke cluster (which may or may not be the OpenShift ClusterID for OpenShift clusters). We will propose to make other changes so that we can get the ClusterID for OpenShift and *KS clusters in a consistent manner. But we can do that elsewhere. We have also captured the ClusterID for the Spoke cluster using the OpenShift ClusterVersionOperator
5. We are trying to see what metrics are gathered at the default 8443 port of the registration controller POD and running into security issues - which we will try to work through.

remove finalizer on rbac when all manifestworks are cleaned (samiliar from pr https://github.com/open-cluster-management/work/pull/23) but i think it is better to move to registration repo.

Fix the below error when running e2e cases before mutatingWebhook Deployment is ready

Taints update check Check the taint to update according to the condition status 
  Should update taints automatically
  /root/go/src/open-cluster-management.io/registration/test/e2e/taint_update_test.go:43

• Failure in Spec Setup (BeforeEach) [0.007 seconds]
Taints update check
/root/go/src/open-cluster-management.io/registration/test/e2e/taint_update_test.go:16
  Check the taint to update according to the condition status
  /root/go/src/open-cluster-management.io/registration/test/e2e/taint_update_test.go:17
    Should update taints automatically [BeforeEach]
    /root/go/src/open-cluster-management.io/registration/test/e2e/taint_update_test.go:43

    Unexpected error:
        <*errors.StatusError | 0xc0003941e0>: {
            ErrStatus: {
                TypeMeta: {Kind: "", APIVersion: ""},
                ListMeta: {
                    SelfLink: "",
                    ResourceVersion: "",
                    Continue: "",
                    RemainingItemCount: nil,
                },
                Status: "Failure",
                Message: "Internal error occurred: failed calling webhook \"managedclustermutators.admission.cluster.open-cluster-management.io\": failed to call webhook: the server is currently unable to handle the request",
                Reason: "InternalError",
                Details: {
                    Name: "",
                    Group: "",
                    Kind: "",
                    UID: "",
                    Causes: [
                        {
                            Type: "",
                            Message: "failed calling webhook \"managedclustermutators.admission.cluster.open-cluster-management.io\": failed to call webhook: the server is currently unable to handle the request",
                            Field: "",
                        },
                    ],
                    RetryAfterSeconds: 0,
                },
                Code: 500,
            },
        }
        Internal error occurred: failed calling webhook "managedclustermutators.admission.cluster.open-cluster-management.io": failed to call webhook: the server is currently unable to handle the request
    occurred

    /root/go/src/open-cluster-management.io/registration/test/e2e/taint_update_test.go:35

Signed-off-by: haoqing0110 [email protected]

Signed-off-by: xuezhaojun [email protected]

Epic: https://github.com/stolostron/backlog/issues/23093

User-story: https://github.com/stolostron/backlog/issues/23929

fix cluster labels update conflict:

E1122 13:49:38.188870       1 base_controller.go:270] "AddOnFeatureDiscoveryController" controller failed to sync "ocm-controlplane-1-mc-4/work-manager", err: Operation cannot be fulfilled on managedclusters.cluster.open-cluster-management.io "ocm-controlplane-1-mc-4": the object has been modified; please apply your changes to the latest version and try again
E1122 13:49:45.423603       1 base_controller.go:270] "AddOnFeatureDiscoveryController" controller failed to sync "ocm-controlplane-1-mc-3/governance-policy-framework", err: Operation cannot be fulfilled on managedclusters.cluster.open-cluster-management.io "ocm-controlplane-1-mc-3": the object has been modified; please apply your changes to the latest version and try again
E1122 13:49:45.438568       1 base_controller.go:270] "AddOnFeatureDiscoveryController" controller failed to sync "ocm-controlplane-1-mc-1/config-policy-controller", err: Operation cannot be fulfilled on managedclusters.cluster.open-cluster-management.io "ocm-controlplane-1-mc-1": the object has been modified; please apply your changes to the latest version and try again
E1122 13:49:45.451226       1 base_controller.go:270] "AddOnFeatureDiscoveryController" controller failed to sync "ocm-controlplane-1-mc-3/config-policy-controller", err: Operation cannot be fulfilled on managedclusters.cluster.open-cluster-management.io "ocm-controlplane-1-mc-3": the object has been modified; please apply your changes to the latest version and try again
E1122 13:49:45.460923       1 base_controller.go:270] "AddOnFeatureDiscoveryController" controller failed to sync "ocm-controlplane-1-mc-1/config-policy-controller", err: Operation cannot be fulfilled on managedclusters.cluster.open-cluster-management.io "ocm-controlplane-1-mc-1": the object has been modified; please apply your changes to the latest version and try again
E1122 13:49:45.477055       1 base_controller.go:270] "AddOnFeatureDiscoveryController" controller failed to sync "ocm-controlplane-1-mc-1/governance-policy-framework", err: Operation cannot be fulfilled on managedclusters.cluster.open-cluster-management.io "ocm-controlplane-1-mc-1": the object has been modified; please apply your changes to the latest version and try again
E1122 13:49:45.488734       1 base_controller.go:270] "AddOnFeatureDiscoveryController" controller failed to sync "ocm-controlplane-1-mc-3", err: Operation cannot be fulfilled on managedclusters.cluster.open-cluster-management.io "ocm-controlplane-1-mc-3": the object has been modified; please apply your changes to the latest version and try again
E1122 13:49:45.503787       1 base_controller.go:270] "AddOnFeatureDiscoveryController" controller failed to sync "ocm-controlplane-1-mc-1/governance-policy-framework", err: Operation cannot be fulfilled on managedclusters.cluster.open-cluster-management.io "ocm-controlplane-1-mc-1": the object has been modified; please apply your changes to the latest version and try again
E1122 13:49:45.511774       1 base_controller.go:270] "AddOnFeatureDiscoveryController" controller failed to sync "ocm-controlplane-1-mc-3", err: Operation cannot be fulfilled on managedclusters.cluster.open-cluster-management.io "ocm-controlplane-1-mc-3": the object has been modified; please apply your changes to the latest version and try again

Signed-off-by: morvencao [email protected]

MicroShift is a project that is exploring how OpenShift and Kubernetes can be optimized for small form factor and edge computing.

• https://microshift.io/docs/home/
• https://microshift.io/docs/getting-started/

microshift is currently recognized as OpenShiftV3 when registering a cluster. This affects Observability to install monitoring stack for microshift.

1. monitor addon/work when deletion
2. delete ns when a special label is not set
3. allow adding other resource to monitor before starting deletion process

Signed-off-by: Jian Qiu [email protected]

We need to add label to all kube resources that are created/managed by registration, thus we can use this label to filter the resources when we inform the resources to reduce the resource usage

For some reason we don't have ability for enabling CSR in our clusters. Can you add another way to authenticate managed clusters without using CSR, like sa-token/secret with kubeconfig?