I keep getting API error from MCS cloud map controller

{"level":"error","ts":1649927224.8055146,"logger":"controllers.Cloudmap","msg":"Cloud Map reconciliation error","error":"operation error ServiceDiscovery: ListServices, exceeded maximum number of attempts, 3, https response error StatusCode: 400, RequestID: 81d0ef9f-ed2b-49d5-98ee-b3b1f6934f7f, api error ThrottlingException: Rate exceeded","stacktrace":"github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/pkg/mod/github.com/go-logr/[email protected]/zapr.go:132\ngithub.com/aws/aws-cloud-map-mcs-controller-for-k8s/pkg/common.logger.Error\n\t/workspace/pkg/common/logger.go:39\ngithub.com/aws/aws-cloud-map-mcs-controller-for-k8s/pkg/controllers.(*CloudMapReconciler).Start\n\t/workspace/pkg/controllers/cloudmap_controller.go:43\nsigs.k8s.io/controller-runtime/pkg/manager.(*controllerManager).startRunnable.func1\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/manager/internal.go:681"}

How can we configure the Cloud map controller to reduce the API call ?

Thanks

We use cloudmap controller in multiple clusters and have the same problem on them. Clusters are linked by vpc peering. Half of the time, an imported service cannot be reached in a pod.

• Service name is correctly resolved in an internal kube adress.
• If I use the endpointslice adress of the remote service, it works on every pod.
• I created a DaemonSet for testing purposes to see if it can be a network/node problem. A pod that have worked on a node can fail after a restart and vice versa.
• We used mcs-controller in v0.2.3 and updated in v0.3.0 and have the same problems.
• Kubernetes 1.21 on all clusters.

I have a feeling that there is some trouble between the service and endpointslice resolution but it don't know how can I debug it.

Controller loads the target region from AWS_REGION env variable, but this variable is never set in configuration files.

https://github.com/aws/aws-cloud-map-mcs-controller-for-k8s/blob/ba9b31ca87969c5bd4e3cdf6259551c9216e36ad/main.go#L90

Error message when running the controller:

2021-10-19T23:07:39.211Z ERROR controllers.ServiceExport error when creating new service in Cloud Map {"serviceexport": "demo/demo-service", "namespace": "demo", "name": "demo-service", "error": "operation error ServiceDiscovery: ListNamespaces, failed to resolve service endpoint, an AWS region is required, but was not found"}

Issue #, if available:

Description of changes: Handle DNS namespaces along-with the HttpNamespace. Create the Service for the DNS namespace type the type SRV. Update the namespaceCache to support namespace struct.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Bumps k8s.io/client-go from 0.24.2 to 0.25.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps k8s.io/apimachinery from 0.24.3 to 0.25.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps k8s.io/api from 0.24.3 to 0.25.2.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps sigs.k8s.io/controller-runtime from 0.12.3 to 0.13.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Implementing feature requested in Issue #100: End-to-end integration test using EKS.

Description of changes: Using EKS, this new end-to-end integration test covers service exports and imports across two EKS clusters, as well as testing service consumption by the importing cluster. The test is largely based on this blog. The test can be run using the command make eks-integration-suite.

This gives the project an integration test that more accurately represents the actual use-case of the controller, in that it tests the controller's functionalities with two EKS clusters instead of one local cluster, as the previous test had been. Adding this to the project workflows allow for more accurate testing of the current state of the controller, ensuring its core functionalities are maintained across commits.

Testing:

• The original integration test,make integration-suite, still runs successfully.
• The test has been successfully run multiple times in a row with no manual intervention
• Should the test fail for whatever reason, the user is able to run make eks-integration-cleanup and the EKS clusters are properly cleaned for any upcoming tests

Next steps:

• This integration test needs be added to project workflow
• The test requires the user to manually setup EKS clusters, a guide which will be made available to the public
• In the future, possible improvements to the test can be making it a one-touch test, so the user does not have to manually create the EKS clusters

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Issue #130

Check to see Go version and dependency versions are compatible. Change go get to go install in line with module dependency changes.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

Installing executables in module mode with go get is deprecated: notice.

/Users/fjywang/aws-cloud-map-mcs-controller-for-k8s/bin/controller-gen "crd:trivialVersions=true,preserveUnknownFields=false" rbac:roleName=manager-role webhook paths="./..." output:crd:artifacts:config=config/crd/bases
bash: /Users/fjywang/aws-cloud-map-mcs-controller-for-k8s/bin/controller-gen: No such file or directory

Current State: Instances correctly deregistered, but dont see the default namespace and services cleaned up outside of that

Desired state: Want the ability to allow the controller to properly clean up all resources it created. The tutorial currently gives cloud map full access, so would follow that still.

Use case: We use SD for other things like ECS services though and eventually want to have a shared namespace controlled by IaC. At that point if we destroyed the stack and there were orphaned services from the k8s controller our IaC would fail to delete the SD namespace

1. Configure 2 kind clusters, and ensure the controller features works across them
2. Inject AWS creds into the controlller docker as env variables
3. Use separate namespace for each run to ensure multiple it can run in parallel

Set aws-region via kustomize edit command instead of the environment variable substitution. Environmental substitution was a bug documented as a feature, and is going to be fixed in the future release. Check this link for more context: https://github.com/kubernetes/website/issues/35669

Also https://kubectl.docs.kubernetes.io/faq/kustomize/eschewedfeatures/#build-time-side-effects-from-cli-args-or-env-variables

Initial implementation of the controller is purely focused on IPv4 support. Research if dual-stack support is needed and implement that.

Dual stack networking is coming to K8s soon: (alpha: "v1.20"/beta: "v1.21"/stable: "v1.23")

See https://github.com/kubernetes/enhancements/tree/master/keps/sig-network/563-dual-stack

In the case that a user is trying to export two services, one of ClusterIP and one of Headless, with the same namespaced name, the controller needs a way to reconcile the conflicting types.

Following the conflict resolution policy, this can be done by propagating the SeviceExport's creationTimestamp to Cloud Map Endpoints as a custom instance attribute, and the older service will take priority, with the newer service not getting exported to Cloud Map.

During benchmark testing, controller completely halted with a panic due to the attempt of accessing an invalid memory address or nil pointer dereference causing a segmentation violation.

Here are the last logs from the manager:

{"level":"info","ts":1655941251.2284162,"logger":"controllers.Cloudmap","msg":"CalculateChanges_ES_Plan","elapsed":9}
{"level":"info","ts":1655941251.228578,"logger":"controllers.Cloudmap","msg":"CalculateChanges_ES_Plan","elapsed":10}
{"level":"info","ts":1655941252.7608504,"logger":"controllers.ServiceExport","msg":"updating Cloud Map service","namespace":"demov2","name":"nginx-benchmark-service-61"}
{"level":"info","ts":1655941252.7608776,"logger":"cloudmap","msg":"fetching a service","namespace":"demov2","name":"nginx-benchmark-service-61"}
{"level":"info","ts":1655941252.76091,"logger":"cloudmap","msg":"creating a new service","namespace":"demov2","name":"nginx-benchmark-service-61"}
{"level":"info","ts":1655941252.8043206,"logger":"cloudmap","msg":"service created","namespace":"demov2","name":"nginx-benchmark-service-61","id":"srv-vvx3saukvkvkytng"}
{"level":"info","ts":1655941252.8043518,"logger":"cloudmap","msg":"fetching a service","namespace":"demov2","name":"nginx-benchmark-service-61"}
{"level":"info","ts":1655941252.8277564,"msg":"Observed a panic in reconciler: runtime error: invalid memory address or nil pointer dereference","controller":"serviceexport","controllerGroup":"multicluster.x-k8s.io","controllerKind":"ServiceExport","serviceExport":{"name":"nginx-benchmark-service-61","namespace":"demov2"},"namespace":"demov2","name":"nginx-benchmark-service-61","reconcileID":"f1c0f919-fdb1-4e63-be72-6aa2f4df35e7"}
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x15204f7]

goroutine 221 [running]:
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile.func1()
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:118 +0x1f4
panic({0x16819e0, 0x271b0e0})
	/usr/local/go/src/runtime/panic.go:1038 +0x215
github.com/aws/aws-cloud-map-mcs-controller-for-k8s/pkg/controllers.(*ServiceExportReconciler).handleUpdate(0xc000461e40, {0x1aad4b0, 0xc0007644c0}, 0xc00032be00, 0xc000d08000)
	/workspace/pkg/controllers/serviceexport_controller.go:136 +0x597
github.com/aws/aws-cloud-map-mcs-controller-for-k8s/pkg/controllers.(*ServiceExportReconciler).Reconcile(0xc000461e40, {0x1aad558, 0xc000b44540}, {{{0xc000d0b876, 0x18}, {0xc000b68780, 0x40d3a7}}})
	/workspace/pkg/controllers/serviceexport_controller.go:93 +0x7d5
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile(0x1aad4b0, {0x1aad558, 0xc000b44540}, {{{0xc000d0b876, 0x17a4d00}, {0xc000b68780, 0x40ee1d}}})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:121 +0xd1
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler(0xc00031b7c0, {0x1aad4b0, 0xc000461c40}, {0x16e0800, 0xc00098b8c0})
	/go/pkg/mod/sigs.k8s.io/controller-ru[email protected]/pkg/internal/controller/controller.go:320 +0x33c
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem(0xc00031b7c0, {0x1aad4b0, 0xc000461c40})
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:273 +0x205
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2()
	/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:234 +0x85
created by sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2
	/go/pkg/mod/sigs.k8s.io/controller-ru[email protected]/pkg/internal/controller/controller.go:230 +0x36f

Note that the code lines do not line up exactly with the Controller's code. serviceexport_controller.go:136 points to the Current: cmService.Endpoints line and serviceexport_controller.go:93 to the return r.handleUpdate(ctx, &serviceExport, &service)