This is a code cleanup PR:

• Removes all the code related to Envoy V2 API. V3 was already the default API so this is the natural step in the deprecation of V2, which has already been removed from https://github.com/envoyproxy/go-control-plane, a dependency of this project.
• Removes the EnvoyBootstrap controller, whose use was deprecated in 0.8.
• Moves the discovery service command code unde cmd/ so the structure of the code is the same for all subcomands.

/kind cleanup /priority important-soon /assign

Context: There is a list of imports in pkg/envoy/serializer/v3/serializer.go that is required so the serialization/deserialization code is able to handle proto messages of the Any type. This is a dynamic list of imports as the files containing protobuffer definitions in go-control-plane can change from version to version. So far this list of imports was manually maintained, which is problematic as the list could end up being out of date.

This PR automates the process of generating the list of imports:

• The list is now maintained in a separate package pkg/envoy/protos/v3 that can be imported from other packages.
• A generator has been written that performs the following tasks:
  • Inspects the project's go.mod to determine the go-control-plane release in use.
  • Clones the specific tag of the go-control-plane repository into memory and looks for the .pb.go files that belong to the v3 api version (though the generator is already able to look for other api versions).
  • Generates the file with all the imports within the pkg/envoy/protos/v3 package.
• go generate is used to trigger the execution of the code generator from the Makefile when the binary is built.

Currently the list of imports is out of date in marin3r-v0.9.0 so a new patch release will be required after this PR as some proto message definitions are missing, resulting in an error if a user tries to use them.

/kind feature /kind bug /priority important-soon /assign

This PR upgrades libs and operator-sdk to latest possible ones, within constrains. Operator SDK manifest generation has been refactored for better maintenance, making it clear where the manifests have been customized within the Kustomize resources. Most of the operator-sdk project scaffolding has been regenerated using the latest version.

Operator SDK has been bumped just to 1.10 as there is an ongoing issue for higher versions, still unresolved even though the issue that was reported is already closed: https://github.com/operator-framework/operator-sdk/issues/5244

/kind feature /priority important-soon /assign

Given that latest releases of envoy have dropped support for the v2 api and that envoy 1.18.3 is currently MARIN3R's default, set v3 as the default API to use.

I also deleted some unused code that was still around.

/kind feature /priority important-soon /assign

This applies only to the V3 protocol.

This impacts the tls_certificate_sds_secret.json config file, not the response sent over the wire to xDS clients.

Marin3r generates this file and adds it to a k8s secret which is mounted into Envoy pods. Envoy uses this file to retrieve the bootstrap certs that allow it to talk to Marin3r over TLS. If there's no "name" field then Envoy v0.17.0 can't process the cert, although older Envoy versions can.

[2021-02-18 23:20:51.419][1][critical][main] [source/server/server.cc:109] error initializing configuration '/etc/envoy/bootstrap/config.json': Proto constraint validation failed (UpstreamTlsContextValidationError.CommonTlsContext: ["embe dded message failed validation"] | caused by CommonTlsContextValidationError.TlsCertificateSdsSecretConfigs[i]: ["embedded message failed validation"] | caused by SdsSecretConfigValidationError.Name: ["value length must be at least " '\x0 1' " runes"]): common_tls_context { tls_certificate_sds_secret_configs { sds_config { path: "/etc/envoy/bootstrap/tls_certificate_sds_secret.json" } } }

[2021-02-18 23:20:51.419][1][info][main] [source/server/server.cc:782] exiting Proto constraint validation failed (UpstreamTlsContextValidationError.CommonTlsContext: ["embedded message failed validation"] | caused by CommonTlsContextValidationError.TlsCertificateSdsSecretConfigs[i]: ["embedded message failed valida tion"] | caused by SdsSecretConfigValidationError.Name: ["value length must be at least " '\x01' " runes"]): common_tls_context { tls_certificate_sds_secret_configs { sds_config { path: "/etc/envoy/bootstrap/tls_certificate_sds_secret.json" } } }

Thank you @roivaz for pointing me to this fix!

'podAffinity' was a poor choice for naming the field because it's basically the affinity field of a Pod spec. It's best if the naming is the same. This field has not make it yet to a stable release, so there is no problem in changing it.

/kind feature /priority important-soon /assign

This PR includes the following:

• Fixed a bug affecting reconcile of EnvoyConfigRevision status: d7687ea600d0293bb62e1146cd195e3e5d631f87
• Implemented a mechanism to internally store statistics related to the xDS protocol messages interchanged between clients and the discovery service: 24e5b59e4636612f11adacf9b7ed5a471d2cc9af
• Reimplemented the self-healing using the internal xDS stats: 1b17645cd666ac48be2d399189b56fe483b9e8d4
• Implemented a backoff algorithm to avoid overloading the envoy clients with retries from the discovery service: 86d1b6dc8ff8aa8f3d19040cebf757f177a37269

/kind feature /priority important-soon /assign

This PR add graceful termination for envoy containers, with connection draining of listeners.

The shutdown manager can be enabled for EnvoyDeployment resources using:

spec:
  shutdownManager: {}

The shutdown manager can be enabled for envoy injected sidecars using the following annotation in Pods:

metadata:
  annotations:
    marin3r.3scale.net/shutdown-manager.enabled: "true"

The shutdown manager is a new command in the Marin3r image that runs a small server and is deployed as a sidecar container to the Envoy container. Container lifecycle hooks are used in both the shutdown manager container and the Envoy container to orchestrate graceful shutdown of the Envoy server, waiting until all the listeners are drained or the 300s timeout is reached.

An example of how the Envoy and the shutdown manager containers are configured:

  containers:
    - name: envoy 
      args:
        - '-c'
        - /etc/envoy/bootstrap/config.json
        - '--service-node'
        - example
        - '--service-cluster'
        - example
        - '--component-log-level'
        - 'config:debug'
      command:
        - envoy
      image: 'envoyproxy/envoy:v1.16.0'
      lifecycle:
        preStop:
          httpGet:
            path: /shutdown
            port: 8090
            scheme: HTTP
      # rest of the container config is omitted
    - name: envoy-shtdn-mgr 
      args:
        - shutdown-manager
        - '--port'
        - '8090'
      image: 'quay.io/3scale/marin3r:v0.8.0-alpha.8'
      lifecycle:
        preStop:
          httpGet:
            path: /drain
            port: 8090
            scheme: HTTP
      # rest of the container config is omitted

/kind feature /priority important-soon

Release v0.10.0.

A small fix has been added to the package generators to wipe the generated file contents before writing.

/kind feature /priority important-soon /assign

This PR allows coordination for extra containers within the Pod with the shutdown manager. A new annotation has been added that allows a user to specify other container names where the shutdown manager lifecycle hook should also be configured. This is a feature that only makes sense for sidecars.

An example of usage:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kuard
  namespace: default
  labels:
    app: kuard
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kuard
  template:
    metadata:
      labels:
        app: kuard
        marin3r.3scale.net/status: "enabled"
      annotations:
        marin3r.3scale.net/node-id: kuard
        marin3r.3scale.net/ports: envoy-https:8443
        marin3r.3scale.net/shutdown-manager.enabled: "true"
        marin3r.3scale.net/shutdown-manager.extra-lifecycle-hooks: kuard
    spec:
      containers:
        - name: kuard
          image: gcr.io/kuar-demo/kuard-amd64:blue
          ports:
            - containerPort: 8080
              name: http
              protocol: TCP

/kind feature /priority importantt-soon /assign

Adds a new event handler in the DiscoveryServiceCertificate controller to trigger a reconcile of any certificate that has its issuer certificate modified. After this, it is safe to enable auto-renewal of the discovery service CA certificate.

/kind feature /assign /priority important-longterm

Note for reviewers: https://github.com/3scale-ops/marin3r/pull/149 needs to be merged first and then I will rebase.

VirtualHost service discovery is available in envoy and go-control-plane already has support for it. Add, if possible, support for it in Marin3r.

/kind feature /priority important-longterm /assign

Why

To use kubernetes endpoints instead of services to improve load balancing decisions.

/kind feature /priority important-longterm /label size/xl /assign

Similarly to #45 we need to add readiness/liveness to the discovery service Deployment. In this case is not sufficient with the endpoints provided by controller-runtime as we also need to assess the health of the discovery service server and somehow aggregate both results in the same endpoint.

Right now the discovery service server runs in a single pod. This is not optimal as if new pods are created while the discovery service pod in down, they will fail. Already running pods are not affected though.

The proposal would be to move the EnvoyConfig controller to the operator pod and leave just the EnvoyConfigRevision discovery service pod. This has some problems that would need to be solved:

• The status would need to have more intelligence as we need to assess that all discovery service pods have synced their cache before declaring an EnvoyConfig cacheStatus as "InSync".

A solution is needed to manage the renewal of the DiscoveryService related certificates:

• The CA
• The server certificate
• The client certificates

Currently all these certificates are just created but never reconciled/renewed so manual action is required to renew them.