How to check the cluster-gateway addon deployed successfully? I have install the cluster-gateway addon and got some info.

Message :"The status of cluster-gateway add-on is unknow" Reason: "ManagedClusterAddOnLeaseNotFound" Status:"Unknow" Type:"Available"

And I got nothing after exec "kubectl get clustergateway".Is there any way to troubleshoot?

Signed-off-by: champly [email protected]

multicluster.NewClusterGatewayRoundTripper not support informer cache. Because informerFactory.Start use ctx listen for stop signals. So add enhance rounttripper include clusterName.

We got a higher latency when using cluster-gateway with ocm-integration=true to access managed cluster k8s resources. And through e2e benchmark test, I found that cluster-gateway will get ManagedCluster per request, and sometimes it will cost X*100ms(sometimes X senconds in the production environment). ManagedCluster is not got from cache，so I add ManagedCluster informer to improve performance, and according to the result of e2e benchmark, the latency is greatly reduced.

add arg --v=4 to gateway-deployment, and then run multicluster e2e benchmark test case test case, I copy the logs of some high latency requests below:

I1206 10:22:12.321915       1 request.go:597] Waited for 392.608513ms due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/cluster.open-cluster-management.io/v1/managedclusters/cluster-81
I1206 10:22:12.326916       1 httplog.go:129] "HTTP" verb="GET" URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster-81/proxy/api/v1/namespaces/kube-system" latency="398.586785ms" userAgent="Go-http-client/2.0" audit-ID="94579a16-45ba-4333-8c83-7bb95dd703ea" srcIP="172.23.211.87:58464" resp=200
I1206 10:22:12.522535       1 request.go:597] Waited for 387.186395ms due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/cluster.open-cluster-management.io/v1/managedclusters/cluster-82
I1206 10:22:12.526706       1 httplog.go:129] "HTTP" verb="GET" URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster-82/proxy/healthz" latency="391.656812ms" userAgent="Go-http-client/2.0" audit-ID="1e02f8f4-8bcd-4fdb-a96d-978074d2f0c5" srcIP="172.23.211.88:36280" resp=200
I1206 10:22:12.722840       1 request.go:597] Waited for 382.323062ms due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/cluster.open-cluster-management.io/v1/managedclusters/cluster-77
I1206 10:22:12.727957       1 httplog.go:129] "HTTP" verb="GET" URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster-77/proxy/api/v1/namespaces/kube-system" latency="387.744393ms" userAgent="Go-http-client/2.0" audit-ID="5022a7df-d6e8-48a7-a53a-2901cc15f187" srcIP="172.23.211.87:58464" resp=200
I1206 10:22:12.922070       1 request.go:597] Waited for 387.643281ms due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/cluster.open-cluster-management.io/v1/managedclusters/cluster-80
I1206 10:22:12.926795       1 httplog.go:129] "HTTP" verb="GET" URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster-80/proxy/healthz" latency="393.948768ms" userAgent="Go-http-client/2.0" audit-ID="ac30211c-7c27-45f2-97b8-c6885826022b" srcIP="172.23.211.88:36280" resp=200
I1206 10:22:13.122193       1 request.go:597] Waited for 392.955174ms due to client-side throttling, not priority and fairness, request: GET:https://10.96.0.1:443/apis/cluster.open-cluster-management.io/v1/managedclusters/cluster-78
I1206 10:22:13.126164       1 httplog.go:129] "HTTP" verb="GET" URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster-78/proxy/api/v1/namespaces/kube-system" latency="397.184533ms" userAgent="Go-http-client/2.0" audit-ID="5a27ffae-58d2-4b63-badf-4c6d832bc8e9" srcIP="172.23.211.87:58464" resp=200

the result of e2e benchmark before adding ManagedCluster informer:

• [MEASUREMENT]
Basic RoundTrip Test
/root/cluster-gateway/e2e/benchmark/configmap.go:32
  list namespace from managed cluster
  /root/cluster-gateway/e2e/benchmark/configmap.go:143

  Ran 10 samples:
  runtime:
    Fastest Time: 0.384s
    Slowest Time: 1.212s
    Average Time: 0.921s ± 0.311s
------------------------------

Ran 1 of 1 Specs in 9.210 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped

You're using deprecated Ginkgo functionality:
=============================================
Ginkgo 2.0 is under active development and will introduce several new features, improvements, and a small handful of breaking changes.
A release candidate for 2.0 is now available and 2.0 should GA in Fall 2021.  Please give the RC a try and send us feedback!
  - To learn more, view the migration guide at https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md
  - For instructions on using the Release Candidate visit https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#using-the-beta
  - To comment, chime in at https://github.com/onsi/ginkgo/issues/711

  Measure is deprecated and will be removed in Ginkgo V2.  Please migrate to gomega/gmeasure.
  Learn more at: https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#removed-measure
    /root/cluster-gateway/e2e/benchmark/configmap.go:143

To silence deprecations that can be silenced set the following environment variable:
  ACK_GINKGO_DEPRECATIONS=1.16.5

--- PASS: TestE2E (9.23s)
PASS
ok  	github.com/oam-dev/cluster-gateway/e2e/benchmark	9.250s

the result of e2e benchmark after adding ManagedCluster informer:

• [MEASUREMENT]
Basic RoundTrip Test
/root/cluster-gateway/e2e/benchmark/configmap.go:32
  list namespace from managed cluster
  /root/cluster-gateway/e2e/benchmark/configmap.go:143

  Ran 10 samples:
  runtime:
    Fastest Time: 0.020s
    Slowest Time: 0.058s
    Average Time: 0.025s ± 0.011s
------------------------------

Ran 1 of 1 Specs in 0.250 seconds
SUCCESS! -- 1 Passed | 0 Failed | 0 Pending | 0 Skipped

You're using deprecated Ginkgo functionality:
=============================================
Ginkgo 2.0 is under active development and will introduce several new features, improvements, and a small handful of breaking changes.
A release candidate for 2.0 is now available and 2.0 should GA in Fall 2021.  Please give the RC a try and send us feedback!
  - To learn more, view the migration guide at https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md
  - For instructions on using the Release Candidate visit https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#using-the-beta
  - To comment, chime in at https://github.com/onsi/ginkgo/issues/711

  Measure is deprecated and will be removed in Ginkgo V2.  Please migrate to gomega/gmeasure.
  Learn more at: https://github.com/onsi/ginkgo/blob/ver2/docs/MIGRATING_TO_V2.md#removed-measure
    /root/cluster-gateway/e2e/benchmark/configmap.go:143

To silence deprecations that can be silenced set the following environment variable:
  ACK_GINKGO_DEPRECATIONS=1.16.5

--- PASS: TestE2E (0.26s)
PASS
ok  	github.com/oam-dev/cluster-gateway/e2e/benchmark	0.286s

Today gateway using cluster credential to forward the request. In multiple tenants, it means all tenants will share same credential and permission to access the target cluster, it is a security risk.

In multiple clusters management, we can assume hub cluster and managed cluster have same structure, for example, create namespace1 for user1 with dev permission on hub cluster, multiple clusters management will create namespace1 on all managed cluster.

With this assumption, to fix the issue above, gateway could do authorization on hub cluster before forward the request, for example, if user1 wants to access namespace1 on managed cluster, then forward, but if user2 wants to access namespace1 on managed cluster, then reject.

feature flag - --proxy-local-authorization=true to enable the change, default is false

Signed-off-by: Yin Da [email protected]

In Kubernetes apiserver lib, the dryRun request for rest.Connector will be filtered which means ClusterGateway cannot make proxy request for dryRun request previously. This PR solves this issue by

1. Escape the dryRun query parameter to __dryRun at the beginning of handling proxy requests.
2. Unescape it while making proxy requests.

https://github.com/kubernetes/kubernetes/blob/b0246c64309ce1ce504ae0a2ca3c66be6ff814e5/staging/src/k8s.io/apiserver/pkg/endpoints/handlers/rest.go#L181

Signed-off-by: Yin Da [email protected]

Implement #120, except:

1. priority is not implemented. The implicit priority for the rules is the lower index rule will have higher priority. Cluster configuration will have higher priority than global configuration.
2. ExternalIdentityExchanger is not implemented, but interfaces are left for future development.

Signed-off-by: Yin Da [email protected]

Add logging arguments for cluster-gateway.

Fixes https://github.com/oam-dev/cluster-gateway/issues/104.

Signed-off-by: Yin Da [email protected]

When --ocm-integration enabled, clusters managed by direct secrets are unavailable to provide services. This PR fixes it.

Proposal: Client Identity Exchanger

Background

In v1.4.0, we have ClientIdentityPenetration feature flag which allows cluster-gateway to pass the hub cluster identity to the managed cluster requests. This ensures that the requests through cluster-gateway to keep consistent identity across all clusters and therefore we can leverage the native Kubernetes RBAC mechanism to deal with the authorization.

Purpose

But there are times that the hub cluster identity is not aligned with the managed cluster identity.

1. For example, in Kubernetes provided by cloud providers, each cluster may have unique ID for its issued identity. Therefore the identity across clusters are not natively consistent. When the problem goes to hybrid cloud, it will be more difficult to manage that.
2. And there are times the user of cluster-gateway may want to use privileges provided by the joined cluster itself, instead of using its own hub cluster identity. This can be seen as a super-user behaviour.

To address the above issues, the ClientIdentityExchanger is proposed here. The ClientIdentityExchanger should be able to exchange identities between the hub cluster and managed clusters depending on the given configuration.

Solution

The ClientIdentityExchanger should work when both ClientIdentityExchanger and ClientIdentityPenetration feature flags are switched on. The ClientIdentityExchanger can be classified into two scopes, one is the global configuration and the other one is the cluster configuration.

1. The global configuration works for all clusters. It can be stored in static files and used by the cluster-gateway. In Kubernetes, we can use ConfigMap/Secret to store it and mount it into the cluster-gateway pod.
2. The cluster configuration works for specific cluster. This data is stored along with the cluster object, such as the cluster.core.oam.dev/client-identity-exchanger-config field in the annotation of cluster secret or OCM managed cluster.

The configuration should be able to describe the rules for exchanging identity, as follows

apiVersion: cluster.core.oam.dev/v1alpha1
kind: ClusterGatewayConfiguration
spec:
  clientIdentityExchanger:
    rules:
    - name: super-user
      source:
        group: sudoer
      type: PrivilegedIdentityExchanger
      priority: 10
    - name: mapping
      source:
        user: user-12345
        cluster: cluster-34567
      target:
        user: user-34567
      type: StaticMappingIdentityExchanger
      priority: 100
    - name: external
      source:
        userPattern: ext-*
      url: http://external.exchanger/
      type: ExternalIdentityExchanger
      priority: 20

We can support three types of exchanger.

• PrivilegedIdentityExchanger will directly use the original identity registered by the cluster.
• StaticMappingIdentityExchanger will make static mappings for the given identity.
• ExternalIdentityExchanger will call external service to provide the identity mapping resources.

OFFLINE means that all the requests come to the offline clusters will be bypass and return normally, and

• vela-core can recognize the defined return code, and set the status for applicaiton cr depending on the defined return code
• vela-apiserver can recognize the defined return code, and return the Rest-API or CLI call depending on the defined return code

当我通过cluster-gateway调用 pod GetLogs接口是会出现连接断开的问题，大概是1分钟左右。 version：v1.3.2 cluster-gateway日志如下： 2022/04/28 13:17:37 httputil: ReverseProxy read error during body copy: context deadline exceeded E0428 13:17:37.307985 1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster1/proxy/api/v1/namespaces/kube-system/pods/kube-apiserver-node1/log?follow=true&tailLines=150" audit-ID="25c2a7a7-7a2f-4bf6-b805-105771b6ebfd" E0428 13:17:37.308822 1 timeout.go:137] post-timeout activity - time-elapsed: 803.726µs, GET "/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster1/proxy/api/v1/namespaces/kube-system/pods/kube-apiserver-node1/log" result: net/http: abort Handler 2022/04/28 13:21:46 httputil: ReverseProxy read error during body copy: context deadline exceeded E0428 13:21:46.937195 1 wrap.go:54] timeout or abort while handling: method=GET URI="/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster1/proxy/api/v1/namespaces/kube-system/pods/kube-apiserver-node1/log?follow=true&tailLines=150" audit-ID="7cd14081-a0d9-4bf7-a85d-a8db310e7b40" E0428 13:21:46.943292 1 timeout.go:137] post-timeout activity - time-elapsed: 6.018041ms, GET "/apis/cluster.core.oam.dev/v1alpha1/clustergateways/cluster1/proxy/api/v1/namespaces/kube-system/pods/kube-apiserver-node1/log" result: net/http: abort Handler