SPIRE is a toolchain of APIs for establishing trust between software systems across a wide variety of hosting platforms

Last update: Jan 2, 2023

Comments: 16

SPIRE (the SPIFFE Runtime Environment) is a toolchain of APIs for establishing trust between software systems across a wide variety of hosting platforms. SPIRE exposes the SPIFFE Workload API, which can attest running software systems and issue SPIFFE IDs and SVIDs to them. This in turn allows two workloads to establish trust between each other, for example by establishing an mTLS connection or by signing and verifying a JWT token. SPIRE can also enable workloads to securely authenticate to a secret store, a database, or a cloud provider service.

Get SPIRE
Learn about SPIRE
Integrate with SPIRE
Contribute to SPIRE
Further Reading
Security

SPIRE is hosted by the Cloud Native Computing Foundation (CNCF) as an incubation-level project. If you are an organization that wants to help shape the evolution of technologies that are container-packaged, dynamically-scheduled and microservices-oriented, consider joining the CNCF.

Get SPIRE

Pre-built releases of SPIRE can be found at https://github.com/spiffe/spire/releases. These releases contain both SPIRE Server and SPIRE Agent binaries.
Alternatively, you can build SPIRE from source.

Learn about SPIRE

Before trying SPIRE, it's a good idea to learn about its architecture and design goals.
Once ready to get started, see the Quickstart Guides for Kubernetes, Linux, and MacOS.
There are several examples demonstrating SPIRE usage in the spire-examples and spire-tutorials repositories.
Check ADOPTERS.md for a list of production SPIRE adopters, a view of the ecosystem, and use cases.
See the SPIRE Roadmap for a list of planned features and enhancements.
Join the SPIFFE community on Slack. If you have any questions about how SPIRE works, or how to get it up and running, the best places to ask questions are the SPIFFE Slack channels.
Download the free book about SPIFFE and SPIRE, "Solving the Bottom Turtle."

Integrate with SPIRE

See Extend SPIRE to learn about the highly extensible SPIRE plugin framework.
Client libraries for interacting with the SPIFFE Workload API are available in Go, Java and C++ languages. See SPIFFE Library Usage Examples for code samples.
SPIRE provides an implementation of the Envoy Secret Discovery Service (SDS) for use with Envoy Proxy. SDS can be used to transparently install and rotate TLS certificates and trust bundles in Envoy. See Using SPIRE with Envoy for more information.

For supported integration versions, see Supported Integrations.

Contribute to SPIRE

The SPIFFE community maintains the SPIRE project. Information on the various SIGs and relevant standards can be found in https://github.com/spiffe/spiffe.

See CONTRIBUTING to get started.
Use GitHub Issues to request features or file bugs.
See GOVERNANCE for SPIFFE and SPIRE governance policies.

Security

Security Assessments

A third party security firm (Cure53) completed a security audit of SPIFFE and SPIRE in February of 2021. Additionally, the CNCF Technical Advisory Group for Security conducted two assessments on SPIFFE and SPIRE in 2018 and 2020. Please find the reports and supporting material, including the threat model exercise results, below.

Reporting Security Vulnerabilities

If you've found a vulnerability or a potential vulnerability in SPIRE please let us know at [email protected]. We'll send a confirmation email to acknowledge your report, and we'll send an additional email when we've identified the issue positively or negatively.

Owner

SPIFFE

Secure Production Identity Framework For Everyone

https://github.com/spiffe/spire https://spiffe.io

Comments

Add Spiffe ID templating for k8s-workload-registrar
Currently the k8s-workload-registrar allows for creation of Spiffe IDs using CRDs

It would be incredibly helpful to be able to have k8s-workload-registrar to allow creating of SpiffeIDs for workloads dynamically using templates defined by a CRD:

apiVersion: spiffeid.spiffe.io/v1beta1 kind: SpiffeID metadata: name: my-spiffe-id namespace: my-namespace spec: spiffeId: spiffe://example.org/{{selector k8s:sa}}/{{selector k8s:pod-uid}} parentId: spiffe://example.org/spire/server
[RFC] Serverless architecture support
[RFC] Serverless architecture support

Co-authored by @MarcosDY.

Background

Serverless computing allows to build applications eliminating the need to manage infrastructure. With serverless applications, the cloud service provider automatically provisions, scales, and manages the infrastructure required to run the code, eliminating the need for server software and hardware management by the developer. The current way of workload attestation in SPIRE does not completely fit in this software design pattern, where the execution context is a temporary runtime environment and is not suitable to have SPIRE Agent running to expose the Workload API alongside the serverless function.

Proposal

In order to allow the issuance of SVIDs to workloads in a serverless environment, we need to provide a way to issue identities to the workload without using the Workload API to obtain an identity. The workload would attest directly to SPIRE Server to obtain its identity. This means that we would go through an attestation process in a similar fashion than node attestation but without yielding an agent role to the attested serverless (and agentless) environment. The attestation process would proceed similarly as the current AttestAgent server RPC, but performed through a new call that would provide an "agentless" identity instead of a node identity in SPIRE. The renewal process would also proceed similarly to the current RenewAgent RPC, where the caller would present an active "agentless" SVID returned by the attestation call or the most recent one from a previous renewal call. This would allow to avoid going through a complete attestation process when the environment already has a valid SVID that needs to be renewed. The criteria to decide if the SVID should be rotated can be similar to the current criteria adopted in SPIRE, i.e.: rotate the SVID if it has less than half of its lifetime left. The proposed solution should facilitate the issuance of identities in a performant manner, focusing on optimizing the usage of resources, otherwise the advantages of the serverless architecture could be seen reduced by the identity issuance process. To that end, this proposal tries to leverage some of the common features available in the cloud providers that aim to solve performance problems, like reusing the execution context if one is available from a previous function call. The proposed process to obtain an identity in a serverless architecture is as follows:

Check if there is already a valid SVID available in the execution context.

If there is no valid SVID, call the "agentless" attestation RPC to get an identity.

Store the obtained identity in a variable declared outside of the function's handler method so it remains initialized, providing additional optimization when the function is invoked again.

If there is already a valid SVID, calculate its lifetime left.

if it has more than half of its lifetime left, just use it.

if it has less than half of its lifetime left, call the renewal RPC and store the obtained identity in a variable declared outside of the function's handler method.

Sample implementation

The following is a description of a sample implementation of the proposed process, including the changes needed in SPIRE and the components required in the serverless environment in order to be able to issue identities without having a SPIRE Agent deployed in the serverless environment.

SPIRE

Add new plugin types to perform the "agentless" attestation in SPIRE Server. Have a new plugin for each provider that has a serverless architecture. For example, there will be a plugin to support AWS Lambda, a plugin for Google Cloud Functions, a plugin for Microsoft Azure Functions and other plugins for any other platform. These are some possible workflows for the implementations:

AWS Lambda: the function signs a GetCallerIdentity query for the AWS Security Token Service (STS) using the AWS Signature v4 algorithm and sends it to SPIRE Server. The credentials used to sign the GetCallerIdentity request come from the AWS Lambda runtime environment variables which avoids the need for an operator to manually provision credentials first. To attest the "agentless" workload, SPIRE Server sends the query to the AWS STS service to validate it and issues an SVID with a SPIFFE ID constructed from attributes extracted from the parsed signed query.

Google Cloud Functions: the function fetches its identity token using the Compute Metadata Server. The attestor plugin in SPIRE Server validates the token provided and issues an SVID with a SPIFFE ID constructed from attributes extracted from the parsed token.

Microsoft Azure: the function obtains its access token from the local token service. The attestor plugin in SPIRE Server validates the token provided and issues an SVID with a SPIFFE ID constructed from attributes extracted from the parsed token.

Attestation data structs are usually shared from github.com/spiffe/spire/pkg/common/plugin/<plugin_name>, which would be inconvenient to consume externally. Instead, the types required could be exposed through Protocol Buffers definitions under the proto/spire hierarchy.

It would be good to expose a library that can be used to facilitate the attestation process from the serverless environment. This library should expose interfaces to construct the attestation material, call the "agentless" attestation RPC in SPIRE Server and ease the reuse of the issued SVID in case that the state of the environment is preserved in a future invocation. It should also provide functionality to perform the SVID renewal process.

Serverless environment

The workload running in the serverless environment needs to be able to be attested without a running SPIRE Agent that exposes the Workload API. Instead, it calls an exposed RPC in SPIRE Server with attestation data that retrieves from the execution runtime. As mentioned above, it would be convenient to have a library that can be consumed in the serverless environment to aid the attestation and identity issuance process. With aim of facilitating the implementation, this proposal recommends implementing a mechanism to externally package dependencies that can be shared across multiple functions. One possible way to achieve this is to have a common interface that can be used to retrieve the identity of the "agentless" workload, that can be called from the running function and is exposed through the runtime environment. For example, in the case of AWS Lambda, the "agentless" attestation functionality can be packaged in a layer. The function that needs to be attested can be configured to use this layer, so it does not need to have it implemented in the function. This layer can also be updated with fixes or improvements without the need of updating the function itself.

Request for Comments

This proposal tries to layout changes needed in SPIRE and possible implementation scenarios to provide support to serverless architectures, focusing on providing a solution for AWS Lambda, Google Cloud Functions and Microsoft Azure Functions. Any feedback on the general direction of this proposal, any missing points, suggestions or thoughts in general is greatly appreciated.
Propose Updated Protos for the SPIRE Server API

The SPIRE registration API was originally written to allow manipulation of SPIRE registration entries. It has since become the de-facto API for all things administrative. This includes decidedly un-registration-y things such as generating join tokens, minting arbitrary SVIDs, agent management, and bundle management.

One problem is simple naming - the "registration api" should not include things that are not related to registration. Another problem is the eventual access control that we want to provide... we currently have the "admin" bit you can set to give carte blanche access, however the scope is too wide. Finally, the registration API has grown organically over time to add features such as ListBy* etc... these features were added in a compatible manner, but we now have a chance to take a new approach to supporting this functionality in a more idiomatic way.

This issue is done when we have loose consensus on the proto(s) for the next iteration of SPIRE management APIs.
spire-server too high CPU usage
Version: 1.2.0

Platform: Linux 5.13.0-30-generic - 20.04.1-Ubuntu SMP x86_64

Subsystem: server

spire-server is running with no agent and without any load on it. CPU consumption is under 1% for ~50-60 minutes, then suddenly it starts consuming ~150-160% CPU and it persists until shutting down the spire-server. The same happens with all replicas. Different environments show different results but the CPU resource consumption is always jumps much higher after similar amount of time spent (~50-60 mins). I observed the above mentioned ~150-160% on a kind cluster, 95-100% on minikube, 100-110% on kvm/qemu and 45-50% on non-virtual k8s environment.

I tried the same with spire-server 1.0.1, 1.1.0 and 1.2.0 images and got the same results. I also created a configuration without k8s-workload-registrar too, but did not help. Most probably it depends on the configuration I use since I have not managed to reproduce it with the reference configuration.

Can you please tell me what is wrong in this configuration? What is the culprit and how can I fix it?

Thanks in advance, Szilard

Failed to collect all selectors for PID" error="workload attestor \"k8s\" failed: rpc error: code = DeadlineExceeded desc = workloadattestor(k8s): no selectors found after max poll attempts"

Version: 1.0.1
Platform: k8s + istio
Subsystem: spire-server:

apiVersion: v1
kind: Namespace
metadata:
  name: spire

---

apiVersion: v1
kind: ServiceAccount
metadata:
  name: spire-server
  namespace: spire

---

apiVersion: v1
kind: Secret
metadata:
  name: spire-server
  namespace: spire
type: Opaque
data:
  bootstrap.key: LS0tLS1CRUdJTiBFQyBQUklWQVRFIEtFWS0tLS0tCk1JR2tBZ0VCQkRBZzJMYnVsWHpRWDFORisyRGkwUkt6TVdmRUdpb0JoaC9mRnB4N3lPRXFrYS8vVHBhZVUzTzUKUUpSWlhkV0hLdWFnQndZRks0RUVBQ0toWkFOaUFBUmFNSDZkSVpMRWhpTE9kdnpqRzdsWVlObVB6U2N2dGJWegpmTi9qeGFITFNacnRqdVlJRXJOOUNTdUFPQzRqaVBSbjdUKzBNZit2eUMwNjBzdXNpbTR6QlllaDdpOXRVRVcxCjdXK1BwZTNwWjRUeVZmQndLOHV6K1p5YTgrcFVyMk09Ci0tLS0tRU5EIEVDIFBSSVZBVEUgS0VZLS0tLS0K

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: spire-server
  namespace: spire
data:
  server.conf: |
    server {
      bind_address = "0.0.0.0"
      bind_port = "8081"
      trust_domain = "example.org"
      data_dir = "/run/spire/data"
      log_level = "DEBUG"
      default_svid_ttl = "1h"
      ca_subject = {
        country = ["US"],
        organization = ["SPIFFE"],
        common_name = "",
      }
    }

    plugins {
      DataStore "sql" {
        plugin_data {
          database_type = "sqlite3"
          connection_string = "/run/spire/data/datastore.sqlite3"
        }
      }

      NodeAttestor "k8s_sat" {
        plugin_data {
          clusters = {
            "demo-cluster" = {
              use_token_review_api_validation = true
              service_account_whitelist = ["spire:spire-agent"]
            }
          }
        }
      }

      NodeResolver "noop" {
        plugin_data {}
      }

      KeyManager "disk" {
        plugin_data {
          keys_path = "/run/spire/data/keys.json"
        }
      }

      UpstreamAuthority "disk" {
        plugin_data {
          key_file_path = "/run/spire/secrets/bootstrap.key"
          cert_file_path = "/run/spire/config/bootstrap.crt"
        }
      }
    }

    health_checks {
      listener_enabled = true
      bind_address = "0.0.0.0"
      bind_port = "8080"
      live_path = "/live"
      ready_path = "/ready"
    }
  bootstrap.crt: |
    -----BEGIN CERTIFICATE-----
    MIIBzDCCAVOgAwIBAgIJAJM4DhRH0vmuMAoGCCqGSM49BAMEMB4xCzAJBgNVBAYT
    AlVTMQ8wDQYDVQQKDAZTUElGRkUwHhcNMTgwNTEzMTkzMzQ3WhcNMjMwNTEyMTkz
    MzQ3WjAeMQswCQYDVQQGEwJVUzEPMA0GA1UECgwGU1BJRkZFMHYwEAYHKoZIzj0C
    AQYFK4EEACIDYgAEWjB+nSGSxIYiznb84xu5WGDZj80nL7W1c3zf48Why0ma7Y7m
    CBKzfQkrgDguI4j0Z+0/tDH/r8gtOtLLrIpuMwWHoe4vbVBFte1vj6Xt6WeE8lXw
    cCvLs/mcmvPqVK9jo10wWzAdBgNVHQ4EFgQUh6XzV6LwNazA+GTEVOdu07o5yOgw
    DwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwGQYDVR0RBBIwEIYOc3Bp
    ZmZlOi8vbG9jYWwwCgYIKoZIzj0EAwQDZwAwZAIwE4Me13qMC9i6Fkx0h26y09QZ
    IbuRqA9puLg9AeeAAyo5tBzRl1YL0KNEp02VKSYJAjBdeJvqjJ9wW55OGj1JQwDF
    D7kWeEB6oMlwPbI/5hEY3azJi16I0uN1JSYTSWGSqWc=
    -----END CERTIFICATE-----

---

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: spire-server
  namespace: spire
  labels:
    app: spire-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: spire-server
  serviceName: spire-server
  template:
    metadata:
      namespace: spire
      labels:
        app: spire-server
    spec:
      serviceAccountName: spire-server
      containers:
        - name: spire-server
          image: xxx/spire-server:1.0.1
          args: ["-config", "/run/spire/config/server.conf"]
          ports:
            - containerPort: 8081
          volumeMounts:
            - name: spire-config
              mountPath: /run/spire/config
              readOnly: true
            - name: spire-secrets
              mountPath: /run/spire/secrets
              readOnly: true
            - name: spire-data
              mountPath: /run/spire/data
              readOnly: false
          livenessProbe:
            httpGet:
              path: /live
              port: 8080
            failureThreshold: 2
            initialDelaySeconds: 15
            periodSeconds: 60
            timeoutSeconds: 3
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
      volumes:
        - name: spire-config
          configMap:
            name: spire-server
        - name: spire-secrets
          secret:
            secretName: spire-server
        - name: spire-data
          emptyDir: {}
---

apiVersion: v1
kind: Service
metadata:
  name: spire-server
  namespace: spire
spec:
  type: ClusterIP
  ports:
    - name: grpc
      port: 8081
      targetPort: 8081
      protocol: TCP
  selector:
    app: spire-server

spire-agent:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: spire-agent
  namespace: spire

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: spire-agent
  namespace: spire
data:
  agent.conf: |
    agent {
      data_dir = "/run/spire"
      log_level = "DEBUG"
      server_address = "spire-server"
      server_port = "8081"
      socket_path = "/run/spire/sockets/agent.sock"
      trust_bundle_path = "/run/spire/config/bootstrap.crt"
      trust_domain = "example.org"
    }

    plugins {
      NodeAttestor "k8s_sat" {
        plugin_data {
          cluster = "demo-cluster"
        }
      }

      KeyManager "memory" {
        plugin_data {
        }
      }

      WorkloadAttestor "k8s" {
        plugin_data {
          # Defaults to the secure kubelet port by default.
          # Minikube does not have a cert in the cluster CA bundle that
          # can authenticate the kubelet cert, so skip validation.
          skip_kubelet_verification = "true"
          #kubelet_read_only_port = "10255"
          node_name_env = "MY_NODE_NAME"
        }
      }

      WorkloadAttestor "unix" {
          plugin_data {
          }
      }
    }

    health_checks {
      listener_enabled = true
      bind_address = "0.0.0.0"
      bind_port = "8080"
      live_path = "/live"
      ready_path = "/ready"
    }
  bootstrap.crt: |
    -----BEGIN CERTIFICATE-----
    MIIBzDCCAVOgAwIBAgIJAJM4DhRH0vmuMAoGCCqGSM49BAMEMB4xCzAJBgNVBAYT
    AlVTMQ8wDQYDVQQKDAZTUElGRkUwHhcNMTgwNTEzMTkzMzQ3WhcNMjMwNTEyMTkz
    MzQ3WjAeMQswCQYDVQQGEwJVUzEPMA0GA1UECgwGU1BJRkZFMHYwEAYHKoZIzj0C
    AQYFK4EEACIDYgAEWjB+nSGSxIYiznb84xu5WGDZj80nL7W1c3zf48Why0ma7Y7m
    CBKzfQkrgDguI4j0Z+0/tDH/r8gtOtLLrIpuMwWHoe4vbVBFte1vj6Xt6WeE8lXw
    cCvLs/mcmvPqVK9jo10wWzAdBgNVHQ4EFgQUh6XzV6LwNazA+GTEVOdu07o5yOgw
    DwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYwGQYDVR0RBBIwEIYOc3Bp
    ZmZlOi8vbG9jYWwwCgYIKoZIzj0EAwQDZwAwZAIwE4Me13qMC9i6Fkx0h26y09QZ
    IbuRqA9puLg9AeeAAyo5tBzRl1YL0KNEp02VKSYJAjBdeJvqjJ9wW55OGj1JQwDF
    D7kWeEB6oMlwPbI/5hEY3azJi16I0uN1JSYTSWGSqWc=
    -----END CERTIFICATE-----

---

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: spire-agent
  namespace: spire
  labels:
    app: spire-agent
spec:
  selector:
    matchLabels:
      app: spire-agent
  template:
    metadata:
      namespace: spire
      labels:
        app: spire-agent
    spec:
      hostPID: true
      hostNetwork: true
      dnsPolicy: ClusterFirstWithHostNet
      serviceAccountName: spire-agent
      initContainers:
        - name: init
          # This is a small image with wait-for-it, choose whatever image
          # you prefer that waits for a service to be up. This image is built
          # from https://github.com/lqhl/wait-for-it
          image: xxx/wait-for-it
          args: ["-t", "30", "spire-server:8081"]
          env:
            - name: MY_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
      containers:
        - name: spire-agent
          image: xxx/spire-agent:1.0.1
          args: ["-config", "/run/spire/config/agent.conf"]
          env:
            - name: MY_NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: status.podIP
          volumeMounts:
            - name: spire-config
              mountPath: /run/spire/config
              readOnly: true
            - name: spire-agent-socket
              mountPath: /run/spire/sockets
              readOnly: false
          livenessProbe:
            httpGet:
              path: /live
              port: 8080
            failureThreshold: 2
            initialDelaySeconds: 15
            periodSeconds: 60
            timeoutSeconds: 3
          readinessProbe:
            httpGet:
              path: /ready
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5
      volumes:
        - name: spire-config
          configMap:
            name: spire-agent
        - name: spire-agent-socket
          hostPath:
            path: /run/spire/sockets
            type: DirectoryOrCreate

When I get the certificate through spire-agent, I get an error. the logs of spire-agent:

time="2022-05-18T11:48:58Z" level=warning msg="Container id not found" attempt=7 container_id=d278326d91c8d21ce00c451ddd1dc0602a054ca96978b08fa7fae2b56cc9a676 external=false plugin_name=k8s plugin_type=WorkloadAttestor retry_interval=500ms subsystem_name=catalog
time="2022-05-18T11:48:58Z" level=warning msg="Container id not found" attempt=8 container_id=d278326d91c8d21ce00c451ddd1dc0602a054ca96978b08fa7fae2b56cc9a676 external=false plugin_name=k8s plugin_type=WorkloadAttestor retry_interval=500ms subsystem_name=catalog
time="2022-05-18T11:48:59Z" level=warning msg="Container id not found" attempt=9 container_id=d278326d91c8d21ce00c451ddd1dc0602a054ca96978b08fa7fae2b56cc9a676 external=false plugin_name=k8s plugin_type=WorkloadAttestor retry_interval=500ms subsystem_name=catalog
time="2022-05-18T11:48:59Z" level=warning msg="Container id not found" attempt=10 container_id=d278326d91c8d21ce00c451ddd1dc0602a054ca96978b08fa7fae2b56cc9a676 external=false plugin_name=k8s plugin_type=WorkloadAttestor retry_interval=500ms subsystem_name=catalog
time="2022-05-18T11:48:59Z" level=error msg="Received error from stream secrets server" error="<nil>" method=StreamSecrets pid=1193 service=SDS.v3 subsystem_name=endpoints
time="2022-05-18T11:48:59Z" level=error msg="Failed to collect all selectors for PID" error="workload attestor \"k8s\" failed: rpc error: code = Canceled desc = workloadattestor(k8s): context canceled" pid=1193 subsystem_name=workload_attestor
time="2022-05-18T11:48:59Z" level=debug msg="PID attested to have selectors" pid=1193 selectors="[type:\"unix\" value:\"uid:1337\" type:\"unix\" value:\"gid:1337\"]" subsystem_name=workload_attestor
time="2022-05-18T11:48:59Z" level=error msg="Failed to attest the workload" error="rpc error: code = Unauthenticated desc = could not verify existence of the original caller: caller is no longer being watched" method=StreamSecrets pid=1193 service=SDS.v3 subsystem_name=endpoints

the logs of spire-server:

time="2022-05-18T11:07:39Z" level=warning msg="Current umask 0022 is too permissive; setting umask 0027"
time="2022-05-18T11:07:39Z" level=info msg="Data directory: \"/run/spire/data\""
time="2022-05-18T11:07:39Z" level=info msg="Opening SQL database" db_type=sqlite3 subsystem_name=sql
time="2022-05-18T11:07:39Z" level=info msg="Initializing new database" subsystem_name=sql
time="2022-05-18T11:07:39Z" level=info msg="Connected to SQL database" read_only=false subsystem_name=sql type=sqlite3 version=3.34.0
time="2022-05-18T11:07:39Z" level=warning msg="The \"noop\" NodeResolver is not required, is deprecated, and will be removed from a future release" subsystem_name=catalog
time="2022-05-18T11:07:39Z" level=warning msg="The `service_account_whitelist` configurable is deprecated and will be removed in a future release. Please use `service_account_allow_list` instead." external=false plugin_name=k8s_sat plugin_type=NodeAttestor subsystem_name=catalog
time="2022-05-18T11:07:39Z" level=info msg="Plugin loaded" external=false plugin_name=k8s_sat plugin_type=NodeAttestor subsystem_name=catalog
time="2022-05-18T11:07:39Z" level=info msg="Plugin loaded" external=false plugin_name=disk plugin_type=KeyManager subsystem_name=catalog
time="2022-05-18T11:07:39Z" level=info msg="Plugin loaded" external=false plugin_name=disk plugin_type=UpstreamAuthority subsystem_name=catalog
time="2022-05-18T11:07:39Z" level=debug msg="Loading journal" path=/run/spire/data/journal.pem subsystem_name=ca_manager
time="2022-05-18T11:07:39Z" level=info msg="Journal loaded" jwt_keys=0 subsystem_name=ca_manager x509_cas=0
time="2022-05-18T11:07:39Z" level=debug msg="Preparing X509 CA" slot=A subsystem_name=ca_manager
time="2022-05-18T11:07:39Z" level=info msg="X509 CA prepared" expiration="2022-05-19T11:07:39Z" issued_at="2022-05-18T11:07:39Z" self_signed=false slot=A subsystem_name=ca_manager
time="2022-05-18T11:07:39Z" level=info msg="X509 CA activated" expiration="2022-05-19T11:07:39Z" issued_at="2022-05-18T11:07:39Z" slot=A subsystem_name=ca_manager
time="2022-05-18T11:07:39Z" level=debug msg="Successfully rotated X.509 CA" subsystem_name=ca_manager trust_domain_id="spiffe://example.org" ttl=86399.559774583
time="2022-05-18T11:07:39Z" level=debug msg="Preparing JWT key" slot=A subsystem_name=ca_manager
time="2022-05-18T11:07:39Z" level=warning msg="UpstreamAuthority plugin does not support JWT-SVIDs. Workloads managed by this server may have trouble communicating with workloads outside this cluster when using JWT-SVIDs." plugin_name=disk subsystem_name=ca_manager
time="2022-05-18T11:07:39Z" level=info msg="JWT key prepared" expiration="2022-05-19T11:07:39Z" issued_at="2022-05-18T11:07:39Z" slot=A subsystem_name=ca_manager
time="2022-05-18T11:07:39Z" level=info msg="JWT key activated" expiration="2022-05-19T11:07:39Z" issued_at="2022-05-18T11:07:39Z" slot=A subsystem_name=ca_manager
time="2022-05-18T11:07:39Z" level=debug msg="Rotating server SVID" subsystem_name=svid_rotator
time="2022-05-18T11:07:39Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:07:39Z" spiffe_id="spiffe://example.org/spire/server" subsystem_name=ca
time="2022-05-18T11:07:39Z" level=info msg="Building in-memory entry cache" subsystem_name=endpoints
time="2022-05-18T11:07:39Z" level=info msg="Completed building in-memory entry cache" subsystem_name=endpoints
time="2022-05-18T11:07:39Z" level=debug msg="Initializing API endpoints" subsystem_name=endpoints
time="2022-05-18T11:07:39Z" level=info msg="Starting TCP server" address="[::]:8081" subsystem_name=endpoints
time="2022-05-18T11:07:39Z" level=info msg="Starting UDS server" address=/tmp/spire-server/private/api.sock subsystem_name=endpoints
time="2022-05-18T11:07:40Z" level=debug msg="Starting checker" name=catalog.datastore subsystem_name=health
time="2022-05-18T11:07:40Z" level=debug msg="Starting checker" name=server.ca subsystem_name=health
time="2022-05-18T11:07:40Z" level=debug msg="Starting checker" name=server.ca.manager subsystem_name=health
time="2022-05-18T11:07:40Z" level=debug msg="Starting checker" name=server subsystem_name=health
time="2022-05-18T11:07:40Z" level=info msg="Serving health checks" address="0.0.0.0:8080" subsystem_name=health
time="2022-05-18T11:07:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:07:47Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/1be11f43-e9e7-41b6-80a4-4dc816b1e889" subsystem_name=ca
time="2022-05-18T11:07:47Z" level=info msg="Agent attestation request completed" address="192.168.1.7:5204" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/1be11f43-e9e7-41b6-80a4-4dc816b1e889" caller_addr="192.168.1.7:5204" method=AttestAgent node_attestor_type=k8s_sat request_id=7bd55951-34d2-4042-b3ac-18f165a0dc9b service=agent.v1.Agent subsystem_name=api
time="2022-05-18T11:07:48Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:07:48Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/64823730-d786-4277-a9bc-106b8eb2f4ff" subsystem_name=ca
time="2022-05-18T11:07:48Z" level=info msg="Agent attestation request completed" address="192.168.1.6:5510" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/64823730-d786-4277-a9bc-106b8eb2f4ff" caller_addr="192.168.1.6:5510" method=AttestAgent node_attestor_type=k8s_sat request_id=76052c90-a828-4a94-a16a-5d6be221e603 service=agent.v1.Agent subsystem_name=api
time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
time="2022-05-18T11:11:17Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:17Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
time="2022-05-18T11:11:32Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:32Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
time="2022-05-18T11:11:32Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:11:32Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
time="2022-05-18T11:13:55Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:13:55Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
time="2022-05-18T11:13:55Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:13:55Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
time="2022-05-18T11:28:36Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:36Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/be1300e4-e93d-40f6-b987-6cb42d20e237" subsystem_name=ca
time="2022-05-18T11:28:36Z" level=info msg="Agent attestation request completed" address="192.168.1.7:26058" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/be1300e4-e93d-40f6-b987-6cb42d20e237" caller_addr="192.168.1.7:26058" method=AttestAgent node_attestor_type=k8s_sat request_id=b00ead65-d001-46e3-89f5-5234442697c9 service=agent.v1.Agent subsystem_name=api
time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
time="2022-05-18T11:28:42Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:42Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/bd4ae7d9-4fbf-4ba0-931c-efc4da0b6e08" subsystem_name=ca
time="2022-05-18T11:28:42Z" level=info msg="Agent attestation request completed" address="192.168.1.6:37898" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/bd4ae7d9-4fbf-4ba0-931c-efc4da0b6e08" caller_addr="192.168.1.6:37898" method=AttestAgent node_attestor_type=k8s_sat request_id=7967a14e-ab5c-4bf9-aaef-21e2d43cf413 service=agent.v1.Agent subsystem_name=api
time="2022-05-18T11:28:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:47Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
time="2022-05-18T11:28:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:47Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
time="2022-05-18T11:28:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:47Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
time="2022-05-18T11:28:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:47Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
time="2022-05-18T11:28:47Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:28:47Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
time="2022-05-18T11:30:53Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:53Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/93fe2161-95cc-4454-84d4-a510616a5dcf" subsystem_name=ca
time="2022-05-18T11:30:53Z" level=info msg="Agent attestation request completed" address="192.168.1.6:42279" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/93fe2161-95cc-4454-84d4-a510616a5dcf" caller_addr="192.168.1.6:42279" method=AttestAgent node_attestor_type=k8s_sat request_id=c3bf5aac-0c6b-4ae5-9fb1-86bcd668f8ec service=agent.v1.Agent subsystem_name=api
time="2022-05-18T11:30:57Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:57Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/ca6a5d08-3517-43a1-8ffc-dc0519a2dea7" subsystem_name=ca
time="2022-05-18T11:30:57Z" level=info msg="Agent attestation request completed" address="192.168.1.7:51286" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/ca6a5d08-3517-43a1-8ffc-dc0519a2dea7" caller_addr="192.168.1.7:51286" method=AttestAgent node_attestor_type=k8s_sat request_id=9dd8a63e-5ace-4381-8a5a-80e2a1e5f0d1 service=agent.v1.Agent subsystem_name=api
time="2022-05-18T11:30:58Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:58Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
time="2022-05-18T11:30:58Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:58Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
time="2022-05-18T11:30:58Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:58Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
time="2022-05-18T11:30:58Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:58Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
time="2022-05-18T11:30:58Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:30:58Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
time="2022-05-18T11:31:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:31:02Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
time="2022-05-18T11:31:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:31:02Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
time="2022-05-18T11:31:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:31:02Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
time="2022-05-18T11:31:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:31:02Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
time="2022-05-18T11:31:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:31:02Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
time="2022-05-18T11:32:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:02Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/32c3378f-b8b0-4df6-b6a0-18fb3bf824f7" subsystem_name=ca
time="2022-05-18T11:32:02Z" level=info msg="Agent attestation request completed" address="192.168.1.7:20470" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/32c3378f-b8b0-4df6-b6a0-18fb3bf824f7" caller_addr="192.168.1.7:20470" method=AttestAgent node_attestor_type=k8s_sat request_id=37844d8e-27f5-42c6-a7be-019bfe6f9c82 service=agent.v1.Agent subsystem_name=api
time="2022-05-18T11:32:02Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:02Z" spiffe_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/7e068ae9-00f4-4786-9191-2b61cb7f08e9" subsystem_name=ca
time="2022-05-18T11:32:02Z" level=info msg="Agent attestation request completed" address="192.168.1.6:24604" agent_id="spiffe://example.org/spire/agent/k8s_sat/demo-cluster/7e068ae9-00f4-4786-9191-2b61cb7f08e9" caller_addr="192.168.1.6:24604" method=AttestAgent node_attestor_type=k8s_sat request_id=60c6024e-eaa6-4f21-8d24-7f4dfff0eb24 service=agent.v1.Agent subsystem_name=api
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/spire/sa/spire-agent" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/default/sa/default" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/test/sa/httpbin" subsystem_name=ca
time="2022-05-18T11:32:07Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:32:07Z" spiffe_id="spiffe://example.org/ns/test/sa/sleep" subsystem_name=ca
time="2022-05-18T11:37:34Z" level=debug msg="Rotating server SVID" subsystem_name=svid_rotator
time="2022-05-18T11:37:34Z" level=debug msg="Signed X509 SVID" expiration="2022-05-18T12:37:34Z" spiffe_id="spiffe://example.org/spire/server" subsystem_name=ca

the agent list in spire-server.

/opt/spire/bin # /opt/spire/bin/spire-server agent list
Found 2 attested agents:

SPIFFE ID         : spiffe://example.org/spire/agent/k8s_sat/demo-cluster/56d7ed58-14ae-4cf5-9880-ffc123b314c1
Attestation type  : k8s_sat
Expiration time   : 2022-05-18 13:02:00 +0000 UTC
Serial number     : 319040273546637254342952962498965679004

SPIFFE ID         : spiffe://example.org/spire/agent/k8s_sat/demo-cluster/63588790-f382-492b-a324-359999d3a394
Attestation type  : k8s_sat
Expiration time   : 2022-05-18 13:02:00 +0000 UTC
Serial number     : 277908708266485423488447531291378329815

the entry in spire-server.

Found 5 entries
Entry ID         : 7ec60e3e-147d-444b-b026-a244279a03c6
SPIFFE ID        : spiffe://example.org/ns/default/sa/default
Parent ID        : spiffe://example.org/ns/spire/sa/spire-agent
Revision         : 0
TTL              : default
Selector         : k8s:ns:default
Selector         : k8s:sa:default

Entry ID         : cc99cf5e-cfd7-44ca-966d-b6313a540447
SPIFFE ID        : spiffe://example.org/ns/istio-system/sa/istio-ingressgateway-service-account
Parent ID        : spiffe://example.org/ns/spire/sa/spire-agent
Revision         : 0
TTL              : default
Selector         : k8s:ns:istio-system
Selector         : k8s:sa:istio-ingressgateway-service-account

Entry ID         : a689cef7-6972-46b1-8277-b3212f31c230
SPIFFE ID        : spiffe://example.org/ns/spire/sa/spire-agent
Parent ID        : spiffe://example.org/spire/server
Revision         : 0
TTL              : default
Selector         : k8s_sat:agent_ns:spire
Selector         : k8s_sat:agent_sa:spire-agent
Selector         : k8s_sat:cluster:demo-cluster

Entry ID         : be9bd775-85cb-4ffa-b914-5493c651d264
SPIFFE ID        : spiffe://example.org/ns/test/sa/httpbin
Parent ID        : spiffe://example.org/ns/spire/sa/spire-agent
Revision         : 0
TTL              : default
Selector         : k8s:ns:test
Selector         : k8s:sa:httpbin

Entry ID         : 3615702c-293e-4d39-9785-16a546c2462b
SPIFFE ID        : spiffe://example.org/ns/test/sa/sleep
Parent ID        : spiffe://example.org/ns/spire/sa/spire-agent
Revision         : 0
TTL              : default
Selector         : k8s:ns:test
Selector         : k8s:sa:sleep

Spire do not federate between k8s clusters which implements Istio multi-primary on different networks

An application is make to federate between two trust domain using spire and Istio. The app being sleep app and helloworld app. The sleep application is able to discover the helloworld service in the other cluster. spire-agent is managing the SDS interface for envoy and its helping to mint the required certs for the envoy. spire is configured with auto attestation of workload.

A sleep app in cluster is able to reach and connect the helloworld app in the same cluster. however when the same sleep application is trying to connect to the helloworld app in the other cluster it fails with SSL error: CERTIFICATE_VERIFY_FAILED

Istio is configured for multi-primary with different network ( meaning there is no direct connection between the pods across cluster boundary). An additional east-west ( ew gw ) is installed. This ew gw has a public address for the clusters to reach and it does SNI pass-through for the traffic to directly reach the service hosted inside the cluster. The service is protected with envoy. Envoy validates all connection for mTLS. on successful validation it would allow the communication to get established.

In case of federation the mTLS will be between two different trust domain and with spire configured correctly it would perform the trust bundle exchange and make the federated CA available to the envoy sitting next to the service.

Expectation: As the CA and federated CA has made available to the helloworld and sleep service, the connection between the sleep application and helloworld should have gone through. While testing it fails and hence this issue has been raised.

Topology

note: entire configuration can be found here: https://github.com/sudeeptoroy/spirefed

brought up two kind clusters: kind-aws-cluster and kind-google-cluster
configured spire for these two clusters and put them in different trust domain: aws.com and google.com
brought up istio on both clusters in multi-primary mode. i have followed this article to bringup istio: https://istio.io/latest/docs/setup/install/multicluster/multi-primary_multi-network/
brought up sample app to rest the federation. 4a. sleep app on aws-cluster 4.b helloworld app on google-cluster 4.c from the sleep app execute curl to helloworld on the other cluster

here is pictorial representation of the topolgy.

Result: The curl fails with this error: kubectl exec --context=kind-aws-cluster -n sample -c sleep sleep-95d8696-bk822 -- curl -sS helloworld.sample:5000/hello upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

Steps to reproduce: Clone: https://github.com/sudeeptoroy/spirefed Follow the readme

Observation:

when i run the helloworld istio-proxy in trace mode i see the following logs:

2022-10-15T10:39:52.419167Z	debug	envoy filter	original_dst: new connection accepted
2022-10-15T10:39:52.419262Z	trace	envoy filter	original_dst: set destination to 10.241.1.8:5000
2022-10-15T10:39:52.419278Z	debug	envoy filter	tls inspector: new connection accepted
2022-10-15T10:39:52.419291Z	trace	envoy filter	tls inspector: recv: 517
2022-10-15T10:39:52.419313Z	trace	envoy filter	tls:onALPN(), ALPN: istio-http/1.1,istio,http/1.1
2022-10-15T10:39:52.419330Z	debug	envoy filter	tls:onServerName(), requestedServerName: outbound_.5000_._.helloworld.sample.svc.cluster.local
2022-10-15T10:39:52.419399Z	trace	envoy misc	enableTimer called on 0x55e2b1ee4080 for 3600000ms, min is 3600000ms
2022-10-15T10:39:52.419448Z	debug	envoy conn_handler	[C146] new connection from 10.241.1.6:58350
2022-10-15T10:39:52.419474Z	trace	envoy connection	[C146] socket event: 3
2022-10-15T10:39:52.419486Z	trace	envoy connection	[C146] write ready
2022-10-15T10:39:52.420578Z	trace	envoy connection	[C146] ssl error occurred while read: WANT_READ
2022-10-15T10:39:52.420705Z	trace	envoy connection	[C146] read ready. dispatch_buffered_data=0
2022-10-15T10:39:52.420744Z	trace	envoy connection	[C146] ssl error occurred while read: WANT_READ
2022-10-15T10:39:52.424542Z	trace	envoy connection	[C146] socket event: 3
2022-10-15T10:39:52.424707Z	trace	envoy connection	[C146] write ready
2022-10-15T10:39:52.424750Z	trace	envoy connection	[C146] ssl error occurred while read: SSL
2022-10-15T10:39:52.424767Z	debug	envoy connection	[C146] TLS error: 268436502:SSL routines:**OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN**
2022-10-15T10:39:52.424780Z	debug	envoy connection	[C146] closing socket: 0
2022-10-15T10:39:52.424824Z	debug	envoy connection	[C146] TLS error: 268436502:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
2022-10-15T10:39:52.424859Z	trace	envoy connection	[C146] raising connection event 0
2022-10-15T10:39:52.424905Z	trace	envoy conn_handler	[C146] connection on event 0
2022-10-15T10:39:52.424918Z	debug	envoy conn_handler	[C146] adding to cleanup list
2022-10-15T10:39:52.424929Z	trace	envoy main	item added to deferred deletion list (size=1)
2022-10-15T10:39:52.424941Z	trace	envoy main	clearing deferred deletion list (size=1)
2022-10-15T10:39:52.432818Z	debug	envoy filter	original_dst: new connection accepted
2022-10-15T10:39:52.432913Z	trace	envoy filter	original_dst: set destination to 10.241.1.8:5000
2022-10-15T10:39:52.432929Z	debug	envoy filter	tls inspector: new connection accepted
2022-10-15T10:39:52.432942Z	trace	envoy filter	tls inspector: recv: 517
2022-10-15T10:39:52.432967Z	trace	envoy filter	tls:onALPN(), ALPN: istio-http/1.1,istio,http/1.1
2022-10-15T10:39:52.432984Z	debug	envoy filter	tls:onServerName(), requestedServerName: outbound_.5000_._.helloworld.sample.svc.cluster.local
2022-10-15T10:39:52.433065Z	trace	envoy misc	enableTimer called on 0x55e2b1ee4080 for 3600000ms, min is 3600000ms
2022-10-15T10:39:52.433086Z	debug	envoy conn_handler	[C147] new connection from 10.241.1.6:58354
2022-10-15T10:39:52.433166Z	trace	envoy connection	[C147] socket event: 3
2022-10-15T10:39:52.433209Z	trace	envoy connection	[C147] write ready
2022-10-15T10:39:52.433493Z	trace	envoy connection	[C147] ssl error occurred while read: WANT_READ
2022-10-15T10:39:52.433679Z	trace	envoy connection	[C147] read ready. dispatch_buffered_data=0
2022-10-15T10:39:52.433698Z	trace	envoy connection	[C147] ssl error occurred while read: WANT_READ
2022-10-15T10:39:52.438301Z	trace	envoy connection	[C147] socket event: 3
2022-10-15T10:39:52.438685Z	trace	envoy connection	[C147] write ready
2022-10-15T10:39:52.438821Z	trace	envoy connection	[C147] ssl error occurred while read: SSL
2022-10-15T10:39:52.438896Z	debug	envoy connection	[C147] TLS error: 268436502:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
2022-10-15T10:39:52.439010Z	debug	envoy connection	[C147] closing socket: 0
2022-10-15T10:39:52.440473Z	debug	envoy connection	[C147] TLS error: 268436502:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
2022-10-15T10:39:52.440731Z	trace	envoy connection	[C147] raising connection event 0
2022-10-15T10:39:52.440776Z	trace	envoy conn_handler	[C147] connection on event 0
2022-10-15T10:39:52.440874Z	debug	envoy conn_handler	[C147] adding to cleanup list
2022-10-15T10:39:52.440920Z	trace	envoy main	item added to deferred deletion list (size=1)
2022-10-15T10:39:52.440939Z	trace	envoy main	clearing deferred deletion list (size=1)
2022-10-15T10:39:52.464980Z	debug	envoy filter	original_dst: new connection accepted
2022-10-15T10:39:52.465038Z	trace	envoy filter	original_dst: set destination to 10.241.1.8:5000
2022-10-15T10:39:52.465045Z	debug	envoy filter	tls inspector: new connection accepted
2022-10-15T10:39:52.465052Z	trace	envoy filter	tls inspector: recv: 517
2022-10-15T10:39:52.465073Z	trace	envoy filter	tls:onALPN(), ALPN: istio-http/1.1,istio,http/1.1
2022-10-15T10:39:52.465083Z	debug	envoy filter	tls:onServerName(), requestedServerName: outbound_.5000_._.helloworld.sample.svc.cluster.local
2022-10-15T10:39:52.465158Z	trace	envoy misc	enableTimer called on 0x55e2b19a1c80 for 3600000ms, min is 3600000ms
2022-10-15T10:39:52.465199Z	debug	envoy conn_handler	[C148] new connection from 10.241.1.6:58360
2022-10-15T10:39:52.465220Z	trace	envoy connection	[C148] socket event: 3
2022-10-15T10:39:52.465224Z	trace	envoy connection	[C148] write ready
2022-10-15T10:39:52.465663Z	trace	envoy connection	[C148] ssl error occurred while read: WANT_READ
2022-10-15T10:39:52.465673Z	trace	envoy connection	[C148] read ready. dispatch_buffered_data=0
2022-10-15T10:39:52.465678Z	trace	envoy connection	[C148] ssl error occurred while read: WANT_READ
2022-10-15T10:39:52.473615Z	trace	envoy connection	[C148] socket event: 3
2022-10-15T10:39:52.473658Z	trace	envoy connection	[C148] write ready
2022-10-15T10:39:52.473751Z	trace	envoy connection	[C148] ssl error occurred while read: SSL
2022-10-15T10:39:52.473760Z	debug	envoy connection	[C148] TLS error: 268436502:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
2022-10-15T10:39:52.473765Z	debug	envoy connection	[C148] closing socket: 0
2022-10-15T10:39:52.473908Z	debug	envoy connection	[C148] TLS error: 268436502:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_UNKNOWN
2022-10-15T10:39:52.473934Z	trace	envoy connection	[C148] raising connection event 0
2022-10-15T10:39:52.473942Z	trace	envoy conn_handler	[C148] connection on event 0
2022-10-15T10:39:52.473944Z	debug	envoy conn_handler	[C148] adding to cleanup list

From the logs and envoy config at helloworld:

The first filter at envoy is "original_dst" where tls inspector should route it to "outbound|5000||helloworld.sample.svc.cluster.local". And for some reason this is not accepting the mTLS from the other domain "aws.com".

Listener dump: check the last section: original_dst

{
  "name": "virtualInbound",
  "active_state": {
    "version_info": "2022-10-15T10:34:48Z/11",
    "listener": {
      "@type": "type.googleapis.com/envoy.config.listener.v3.Listener",
      "name": "virtualInbound",
      "address": {
        "socket_address": {
          "address": "0.0.0.0",
          "port_value": 15006
        }
      },
      "filter_chains": [
        {
          "filter_chain_match": {
            "destination_port": 15006
          },
          "filters": [
            {
              "name": "istio.metadata_exchange",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
                "protocol": "istio-peer-exchange"
              }
            },
            {
              "name": "istio.stats",
              "typed_config": {
                "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                "type_url": "type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm",
                "value": {
                  "config": {
                    "root_id": "stats_inbound",
                    "vm_config": {
                      "vm_id": "tcp_stats_inbound",
                      "runtime": "envoy.wasm.runtime.null",
                      "code": {
                        "local": {
                          "inline_string": "envoy.wasm.stats"
                        }
                      }
                    },
                    "configuration": {
                      "@type": "type.googleapis.com/google.protobuf.StringValue",
                      "value": "{\n  \"debug\": \"false\",\n  \"stat_prefix\": \"istio\",\n  \"metrics\": [\n    {\n      \"dimensions\": {\n        \"destination_cluster\": \"node.metadata['CLUSTER_ID']\",\n        \"source_cluster\": \"downstream_peer.cluster_id\"\n      }\n    }\n  ]\n}\n"
                    }
                  }
                }
              }
            },
            {
              "name": "envoy.filters.network.tcp_proxy",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
                "stat_prefix": "BlackHoleCluster",
                "cluster": "BlackHoleCluster"
              }
            }
          ],
          "name": "virtualInbound-blackhole"
        },
        {
          "filter_chain_match": {
            "prefix_ranges": [
              {
                "address_prefix": "0.0.0.0",
                "prefix_len": 0
              }
            ],
            "transport_protocol": "tls",
            "application_protocols": [
              "istio-http/1.0",
              "istio-http/1.1",
              "istio-h2"
            ]
          },
          "filters": [
            {
              "name": "istio.metadata_exchange",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
                "protocol": "istio-peer-exchange"
              }
            },
            {
              "name": "envoy.filters.network.http_connection_manager",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                "stat_prefix": "InboundPassthroughClusterIpv4",
                "route_config": {
                  "name": "InboundPassthroughClusterIpv4",
                  "virtual_hosts": [
                    {
                      "name": "inbound|http|0",
                      "domains": [
                        "*"
                      ],
                      "routes": [
                        {
                          "match": {
                            "prefix": "/"
                          },
                          "route": {
                            "cluster": "InboundPassthroughClusterIpv4",
                            "timeout": "0s",
                            "max_stream_duration": {
                              "max_stream_duration": "0s",
                              "grpc_timeout_header_max": "0s"
                            }
                          },
                          "decorator": {
                            "operation": ":0/*"
                          },
                          "name": "default"
                        }
                      ]
                    }
                  ],
                  "validate_clusters": false
                },
                "http_filters": [
                  {
                    "name": "istio.metadata_exchange",
                    "typed_config": {
                      "@type": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
                      "config": {
                        "vm_config": {
                          "runtime": "envoy.wasm.runtime.null",
                          "code": {
                            "local": {
                              "inline_string": "envoy.wasm.metadata_exchange"
                            }
                          }
                        },
                        "configuration": {
                          "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange"
                        }
                      }
                    }
                  },
                  {
                    "name": "envoy.filters.http.fault",
                    "typed_config": {
                      "@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
                    }
                  },
                  {
                    "name": "envoy.filters.http.cors",
                    "typed_config": {
                      "@type": "type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors"
                    }
                  },
                  {
                    "name": "istio.stats",
                    "typed_config": {
                      "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                      "type_url": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
                      "value": {
                        "config": {
                          "root_id": "stats_inbound",
                          "vm_config": {
                            "vm_id": "stats_inbound",
                            "runtime": "envoy.wasm.runtime.null",
                            "code": {
                              "local": {
                                "inline_string": "envoy.wasm.stats"
                              }
                            }
                          },
                          "configuration": {
                            "@type": "type.googleapis.com/google.protobuf.StringValue",
                            "value": "{\n  \"debug\": \"false\",\n  \"stat_prefix\": \"istio\",\n  \"disable_host_header_fallback\": true,\n  \"metrics\": [\n    {\n      \"dimensions\": {\n        \"destination_cluster\": \"node.metadata['CLUSTER_ID']\",\n        \"source_cluster\": \"downstream_peer.cluster_id\"\n      }\n    }\n  ]\n}\n"
                          }
                        }
                      }
                    }
                  },
                  {
                    "name": "envoy.filters.http.router",
                    "typed_config": {
                      "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
                    }
                  }
                ],
                "tracing": {
                  "client_sampling": {
                    "value": 100
                  },
                  "random_sampling": {
                    "value": 1
                  },
                  "overall_sampling": {
                    "value": 100
                  },
                  "custom_tags": [
                    {
                      "tag": "istio.authorization.dry_run.allow_policy.name",
                      "metadata": {
                        "kind": {
                          "request": {}
                        },
                        "metadata_key": {
                          "key": "envoy.filters.http.rbac",
                          "path": [
                            {
                              "key": "istio_dry_run_allow_shadow_effective_policy_id"
                            }
                          ]
                        }
                      }
                    },
                    {
                      "tag": "istio.authorization.dry_run.allow_policy.result",
                      "metadata": {
                        "kind": {
                          "request": {}
                        },
                        "metadata_key": {
                          "key": "envoy.filters.http.rbac",
                          "path": [
                            {
                              "key": "istio_dry_run_allow_shadow_engine_result"
                            }
                          ]
                        }
                      }
                    },
                    {
                      "tag": "istio.authorization.dry_run.deny_policy.name",
                      "metadata": {
                        "kind": {
                          "request": {}
                        },
                        "metadata_key": {
                          "key": "envoy.filters.http.rbac",
                          "path": [
                            {
                              "key": "istio_dry_run_deny_shadow_effective_policy_id"
                            }
                          ]
                        }
                      }
                    },
                    {
                      "tag": "istio.authorization.dry_run.deny_policy.result",
                      "metadata": {
                        "kind": {
                          "request": {}
                        },
                        "metadata_key": {
                          "key": "envoy.filters.http.rbac",
                          "path": [
                            {
                              "key": "istio_dry_run_deny_shadow_engine_result"
                            }
                          ]
                        }
                      }
                    },
                    {
                      "tag": "istio.canonical_revision",
                      "literal": {
                        "value": "v2"
                      }
                    },
                    {
                      "tag": "istio.canonical_service",
                      "literal": {
                        "value": "helloworld"
                      }
                    },
                    {
                      "tag": "istio.mesh_id",
                      "literal": {
                        "value": "devup-mesh"
                      }
                    },
                    {
                      "tag": "istio.namespace",
                      "literal": {
                        "value": "sample"
                      }
                    }
                  ]
                },
                "server_name": "istio-envoy",
                "use_remote_address": false,
                "forward_client_cert_details": "APPEND_FORWARD",
                "set_current_client_cert_details": {
                  "subject": true,
                  "dns": true,
                  "uri": true
                },
                "upgrade_configs": [
                  {
                    "upgrade_type": "websocket"
                  }
                ],
                "stream_idle_timeout": "0s",
                "normalize_path": true,
                "request_id_extension": {
                  "typed_config": {
                    "@type": "type.googleapis.com/envoy.extensions.request_id.uuid.v3.UuidRequestIdConfig",
                    "use_request_id_for_trace_sampling": true
                  }
                },
                "path_with_escaped_slashes_action": "KEEP_UNCHANGED"
              }
            }
          ],
          "transport_socket": {
            "name": "envoy.transport_sockets.tls",
            "typed_config": {
              "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
              "common_tls_context": {
                "tls_params": {
                  "tls_minimum_protocol_version": "TLSv1_2",
                  "tls_maximum_protocol_version": "TLSv1_3",
                  "cipher_suites": [
                    "ECDHE-ECDSA-AES256-GCM-SHA384",
                    "ECDHE-RSA-AES256-GCM-SHA384",
                    "ECDHE-ECDSA-AES128-GCM-SHA256",
                    "ECDHE-RSA-AES128-GCM-SHA256",
                    "AES256-GCM-SHA384",
                    "AES128-GCM-SHA256"
                  ]
                },
                "alpn_protocols": [
                  "h2",
                  "http/1.1"
                ],
                "tls_certificate_sds_secret_configs": [
                  {
                    "name": "default",
                    "sds_config": {
                      "api_config_source": {
                        "api_type": "GRPC",
                        "grpc_services": [
                          {
                            "envoy_grpc": {
                              "cluster_name": "sds-grpc"
                            }
                          }
                        ],
                        "set_node_on_first_message_only": true,
                        "transport_api_version": "V3"
                      },
                      "initial_fetch_timeout": "0s",
                      "resource_api_version": "V3"
                    }
                  }
                ],
                "combined_validation_context": {
                  "default_validation_context": {
                    "match_subject_alt_names": [
                      {
                        "prefix": "spiffe://google.com/"
                      }
                    ]
                  },
                  "validation_context_sds_secret_config": {
                    "name": "ROOTCA",
                    "sds_config": {
                      "api_config_source": {
                        "api_type": "GRPC",
                        "grpc_services": [
                          {
                            "envoy_grpc": {
                              "cluster_name": "sds-grpc"
                            }
                          }
                        ],
                        "set_node_on_first_message_only": true,
                        "transport_api_version": "V3"
                      },
                      "initial_fetch_timeout": "0s",
                      "resource_api_version": "V3"
                    }
                  }
                }
              },
              "require_client_certificate": true
            }
          },
          "name": "virtualInbound-catchall-http"
        },
        {
          "filter_chain_match": {
            "prefix_ranges": [
              {
                "address_prefix": "0.0.0.0",
                "prefix_len": 0
              }
            ],
            "transport_protocol": "tls"
          },
          "filters": [
            {
              "name": "istio.metadata_exchange",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
                "protocol": "istio-peer-exchange"
              }
            },
            {
              "name": "istio.stats",
              "typed_config": {
                "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                "type_url": "type.googleapis.com/envoy.extensions.filters.network.wasm.v3.Wasm",
                "value": {
                  "config": {
                    "root_id": "stats_inbound",
                    "vm_config": {
                      "vm_id": "tcp_stats_inbound",
                      "runtime": "envoy.wasm.runtime.null",
                      "code": {
                        "local": {
                          "inline_string": "envoy.wasm.stats"
                        }
                      }
                    },
                    "configuration": {
                      "@type": "type.googleapis.com/google.protobuf.StringValue",
                      "value": "{\n  \"debug\": \"false\",\n  \"stat_prefix\": \"istio\",\n  \"metrics\": [\n    {\n      \"dimensions\": {\n        \"destination_cluster\": \"node.metadata['CLUSTER_ID']\",\n        \"source_cluster\": \"downstream_peer.cluster_id\"\n      }\n    }\n  ]\n}\n"
                    }
                  }
                }
              }
            },
            {
              "name": "envoy.filters.network.tcp_proxy",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
                "stat_prefix": "InboundPassthroughClusterIpv4",
                "cluster": "InboundPassthroughClusterIpv4"
              }
            }
          ],
          "transport_socket": {
            "name": "envoy.transport_sockets.tls",
            "typed_config": {
              "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
              "common_tls_context": {
                "tls_params": {
                  "tls_minimum_protocol_version": "TLSv1_2",
                  "tls_maximum_protocol_version": "TLSv1_3",
                  "cipher_suites": [
                    "ECDHE-ECDSA-AES256-GCM-SHA384",
                    "ECDHE-RSA-AES256-GCM-SHA384",
                    "ECDHE-ECDSA-AES128-GCM-SHA256",
                    "ECDHE-RSA-AES128-GCM-SHA256",
                    "AES256-GCM-SHA384",
                    "AES128-GCM-SHA256"
                  ]
                },
                "alpn_protocols": [
                  "istio-peer-exchange",
                  "h2",
                  "http/1.1"
                ],
                "tls_certificate_sds_secret_configs": [
                  {
                    "name": "default",
                    "sds_config": {
                      "api_config_source": {
                        "api_type": "GRPC",
                        "grpc_services": [
                          {
                            "envoy_grpc": {
                              "cluster_name": "sds-grpc"
                            }
                          }
                        ],
                        "set_node_on_first_message_only": true,
                        "transport_api_version": "V3"
                      },
                      "initial_fetch_timeout": "0s",
                      "resource_api_version": "V3"
                    }
                  }
                ],
                "combined_validation_context": {
                  "default_validation_context": {
                    "match_subject_alt_names": [
                      {
                        "prefix": "spiffe://google.com/"
                      }
                    ]
                  },
                  "validation_context_sds_secret_config": {
                    "name": "ROOTCA",
                    "sds_config": {
                      "api_config_source": {
                        "api_type": "GRPC",
                        "grpc_services": [
                          {
                            "envoy_grpc": {
                              "cluster_name": "sds-grpc"
                            }
                          }
                        ],
                        "set_node_on_first_message_only": true,
                        "transport_api_version": "V3"
                      },
                      "initial_fetch_timeout": "0s",
                      "resource_api_version": "V3"
                    }
                  }
                }
              },
              "require_client_certificate": true
            }
          },
          "name": "virtualInbound"
        },
        {
          "filter_chain_match": {
            "destination_port": 5000,
            "transport_protocol": "tls"
          },
          "filters": [
            {
              "name": "istio.metadata_exchange",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
                "protocol": "istio-peer-exchange"
              }
            },
            {
              "name": "envoy.filters.network.http_connection_manager",
              "typed_config": {
                "@type": "type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager",
                "stat_prefix": "inbound_0.0.0.0_5000",
                "route_config": {
                  "name": "inbound|5000||",
                  "virtual_hosts": [
                    {
                      "name": "inbound|http|5000",
                      "domains": [
                        "*"
                      ],
                      "routes": [
                        {
                          "match": {
                            "prefix": "/"
                          },
                          "route": {
                            "cluster": "inbound|5000||",
                            "timeout": "0s",
                            "max_stream_duration": {
                              "max_stream_duration": "0s",
                              "grpc_timeout_header_max": "0s"
                            }
                          },
                          "decorator": {
                            "operation": "helloworld.sample.svc.cluster.local:5000/*"
                          },
                          "name": "default"
                        }
                      ]
                    }
                  ],
                  "validate_clusters": false
                },
                "http_filters": [
                  {
                    "name": "istio.metadata_exchange",
                    "typed_config": {
                      "@type": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
                      "config": {
                        "vm_config": {
                          "runtime": "envoy.wasm.runtime.null",
                          "code": {
                            "local": {
                              "inline_string": "envoy.wasm.metadata_exchange"
                            }
                          }
                        },
                        "configuration": {
                          "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange"
                        }
                      }
                    }
                  },
                  {
                    "name": "envoy.filters.http.fault",
                    "typed_config": {
                      "@type": "type.googleapis.com/envoy.extensions.filters.http.fault.v3.HTTPFault"
                    }
                  },
                  {
                    "name": "envoy.filters.http.cors",
                    "typed_config": {
                      "@type": "type.googleapis.com/envoy.extensions.filters.http.cors.v3.Cors"
                    }
                  },
                  {
                    "name": "istio.stats",
                    "typed_config": {
                      "@type": "type.googleapis.com/udpa.type.v1.TypedStruct",
                      "type_url": "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm",
                      "value": {
                        "config": {
                          "root_id": "stats_inbound",
                          "vm_config": {
                            "vm_id": "stats_inbound",
                            "runtime": "envoy.wasm.runtime.null",
                            "code": {
                              "local": {
                                "inline_string": "envoy.wasm.stats"
                              }
                            }
                          },
                          "configuration": {
                            "@type": "type.googleapis.com/google.protobuf.StringValue",
                            "value": "{\n  \"debug\": \"false\",\n  \"stat_prefix\": \"istio\",\n  \"disable_host_header_fallback\": true,\n  \"metrics\": [\n    {\n      \"dimensions\": {\n        \"destination_cluster\": \"node.metadata['CLUSTER_ID']\",\n        \"source_cluster\": \"downstream_peer.cluster_id\"\n      }\n    }\n  ]\n}\n"
                          }
                        }
                      }
                    }
                  },
                  {
                    "name": "envoy.filters.http.router",
                    "typed_config": {
                      "@type": "type.googleapis.com/envoy.extensions.filters.http.router.v3.Router"
                    }
                  }
                ],
                "tracing": {
                  "client_sampling": {
                    "value": 100
                  },
                  "random_sampling": {
                    "value": 1
                  },
                  "overall_sampling": {
                    "value": 100
                  },
                  "custom_tags": [
                    {
                      "tag": "istio.authorization.dry_run.allow_policy.name",
                      "metadata": {
                        "kind": {
                          "request": {}
                        },
                        "metadata_key": {
                          "key": "envoy.filters.http.rbac",
                          "path": [
                            {
                              "key": "istio_dry_run_allow_shadow_effective_policy_id"
                            }
                          ]
                        }
                      }
                    },
                    {
                      "tag": "istio.authorization.dry_run.allow_policy.result",
                      "metadata": {
                        "kind": {
                          "request": {}
                        },
                        "metadata_key": {
                          "key": "envoy.filters.http.rbac",
                          "path": [
                            {
                              "key": "istio_dry_run_allow_shadow_engine_result"
                            }
                          ]
                        }
                      }
                    },
                    {
                      "tag": "istio.authorization.dry_run.deny_policy.name",
                      "metadata": {
                        "kind": {
                          "request": {}
                        },
                        "metadata_key": {
                          "key": "envoy.filters.http.rbac",
                          "path": [
                            {
                              "key": "istio_dry_run_deny_shadow_effective_policy_id"
                            }
                          ]
                        }
                      }
                    },
                    {
                      "tag": "istio.authorization.dry_run.deny_policy.result",
                      "metadata": {
                        "kind": {
                          "request": {}
                        },
                        "metadata_key": {
                          "key": "envoy.filters.http.rbac",
                          "path": [
                            {
                              "key": "istio_dry_run_deny_shadow_engine_result"
                            }
                          ]
                        }
                      }
                    },
                    {
                      "tag": "istio.canonical_revision",
                      "literal": {
                        "value": "v2"
                      }
                    },
                    {
                      "tag": "istio.canonical_service",
                      "literal": {
                        "value": "helloworld"
                      }
                    },
                    {
                      "tag": "istio.mesh_id",
                      "literal": {
                        "value": "devup-mesh"
                      }
                    },
                    {
                      "tag": "istio.namespace",
                      "literal": {
                        "value": "sample"
                      }
                    }
                  ]
                },
                "server_name": "istio-envoy",
                "use_remote_address": false,
                "forward_client_cert_details": "APPEND_FORWARD",
                "set_current_client_cert_details": {
                  "subject": true,
                  "dns": true,
                  "uri": true
                },
                "upgrade_configs": [
                  {
                    "upgrade_type": "websocket"
                  }
                ],
                "stream_idle_timeout": "0s",
                "normalize_path": true,
                "request_id_extension": {
                  "typed_config": {
                    "@type": "type.googleapis.com/envoy.extensions.request_id.uuid.v3.UuidRequestIdConfig",
                    "use_request_id_for_trace_sampling": true
                  }
                },
                "path_with_escaped_slashes_action": "KEEP_UNCHANGED"
              }
            }
          ],
          "transport_socket": {
            "name": "envoy.transport_sockets.tls",
            "typed_config": {
              "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext",
              "common_tls_context": {
                "tls_params": {
                  "tls_minimum_protocol_version": "TLSv1_2",
                  "tls_maximum_protocol_version": "TLSv1_3",
                  "cipher_suites": [
                    "ECDHE-ECDSA-AES256-GCM-SHA384",
                    "ECDHE-RSA-AES256-GCM-SHA384",
                    "ECDHE-ECDSA-AES128-GCM-SHA256",
                    "ECDHE-RSA-AES128-GCM-SHA256",
                    "AES256-GCM-SHA384",
                    "AES128-GCM-SHA256"
                  ]
                },
                "alpn_protocols": [
                  "h2",
                  "http/1.1"
                ],
                "tls_certificate_sds_secret_configs": [
                  {
                    "name": "default",
                    "sds_config": {
                      "api_config_source": {
                        "api_type": "GRPC",
                        "grpc_services": [
                          {
                            "envoy_grpc": {
                              "cluster_name": "sds-grpc"
                            }
                          }
                        ],
                        "set_node_on_first_message_only": true,
                        "transport_api_version": "V3"
                      },
                      "initial_fetch_timeout": "0s",
                      "resource_api_version": "V3"
                    }
                  }
                ],
                "combined_validation_context": {
                  "default_validation_context": {
                    "match_subject_alt_names": [
                      {
                        "prefix": "spiffe://google.com/"
                      }
                    ]
                  },
                  "validation_context_sds_secret_config": {
                    "name": "ROOTCA",
                    "sds_config": {
                      "api_config_source": {
                        "api_type": "GRPC",
                        "grpc_services": [
                          {
                            "envoy_grpc": {
                              "cluster_name": "sds-grpc"
                            }
                          }
                        ],
                        "set_node_on_first_message_only": true,
                        "transport_api_version": "V3"
                      },
                      "initial_fetch_timeout": "0s",
                      "resource_api_version": "V3"
                    }
                  }
                }
              },
              "require_client_certificate": true
            }
          },
          "name": "0.0.0.0_5000"
        }
      ],
      "listener_filters": [
        {
          "name": "envoy.filters.listener.original_dst",
          "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.filters.listener.original_dst.v3.OriginalDst"
          }
        },
        {
          "name": "envoy.filters.listener.tls_inspector",
          "typed_config": {
            "@type": "type.googleapis.com/envoy.extensions.filters.listener.tls_inspector.v3.TlsInspector"
          },
          "filter_disabled": {
            "destination_port_range": {
              "start": 15006,
              "end": 15007
            }
          }
        }
      ],
      "listener_filters_timeout": "0s",
      "traffic_direction": "INBOUND",
      "continue_on_listener_filters_timeout": true
    },
    "last_updated": "2022-10-15T10:35:30.395Z"
  }
}

Cluster dump: this is the cluster dump which shows that the SDS validation is with ROOTCA

"outbound|5000||helloworld.sample.svc.cluster.local"
{
  "@type": "type.googleapis.com/envoy.config.cluster.v3.Cluster",
  "name": "outbound|5000||helloworld.sample.svc.cluster.local",
  "type": "EDS",
  "eds_cluster_config": {
    "eds_config": {
      "ads": {},
      "initial_fetch_timeout": "0s",
      "resource_api_version": "V3"
    },
    "service_name": "outbound|5000||helloworld.sample.svc.cluster.local"
  },
  "connect_timeout": "10s",
  "lb_policy": "LEAST_REQUEST",
  "circuit_breakers": {
    "thresholds": [
      {
        "max_connections": 4294967295,
        "max_pending_requests": 4294967295,
        "max_requests": 4294967295,
        "max_retries": 4294967295,
        "track_remaining": true
      }
    ]
  },
  "metadata": {
    "filter_metadata": {
      "istio": {
        "default_original_port": 5000,
        "services": [
          {
            "name": "helloworld",
            "host": "helloworld.sample.svc.cluster.local",
            "namespace": "sample"
          }
        ]
      }
    }
  },
  "common_lb_config": {
    "locality_weighted_lb_config": {}
  },
  "filters": [
    {
      "name": "istio.metadata_exchange",
      "typed_config": {
        "@type": "type.googleapis.com/envoy.tcp.metadataexchange.config.MetadataExchange",
        "protocol": "istio-peer-exchange"
      }
    }
  ],
  "transport_socket_matches": [
    {
      "name": "tlsMode-istio",
      "match": {
        "tlsMode": "istio"
      },
      "transport_socket": {
        "name": "envoy.transport_sockets.tls",
        "typed_config": {
          "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext",
          "common_tls_context": {
            "tls_params": {
              "tls_minimum_protocol_version": "TLSv1_2",
              "tls_maximum_protocol_version": "TLSv1_3"
            },
            "alpn_protocols": [
              "istio-peer-exchange",
              "istio"
            ],
            "tls_certificate_sds_secret_configs": [
              {
                "name": "default",
                "sds_config": {
                  "api_config_source": {
                    "api_type": "GRPC",
                    "grpc_services": [
                      {
                        "envoy_grpc": {
                          "cluster_name": "sds-grpc"
                        }
                      }
                    ],
                    "set_node_on_first_message_only": true,
                    "transport_api_version": "V3"
                  },
                  "initial_fetch_timeout": "0s",
                  "resource_api_version": "V3"
                }
              }
            ],
            "combined_validation_context": {
              "default_validation_context": {
                "match_subject_alt_names": [
                  {
                    "exact": "spiffe://google.com/ns/sample/sa/default"
                  }
                ]
              },
              "validation_context_sds_secret_config": {
                "name": "ROOTCA",
                "sds_config": {
                  "api_config_source": {
                    "api_type": "GRPC",
                    "grpc_services": [
                      {
                        "envoy_grpc": {
                          "cluster_name": "sds-grpc"
                        }
                      }
                    ],
                    "set_node_on_first_message_only": true,
                    "transport_api_version": "V3"
                  },
                  "initial_fetch_timeout": "0s",
                  "resource_api_version": "V3"
                }
              }
            }
          },
          "sni": "outbound_.5000_._.helloworld.sample.svc.cluster.local"
        }
      }
    },
    {
      "name": "tlsMode-disabled",
      "match": {},
      "transport_socket": {
        "name": "envoy.transport_sockets.raw_buffer",
        "typed_config": {
          "@type": "type.googleapis.com/envoy.extensions.transport_sockets.raw_buffer.v3.RawBuffer"
        }
      }
    }
  ]
}

Secret dump: secret dump which has both the CA configured. upon inspecting you would see that they are minded by spire.


  {
   "@type": "type.googleapis.com/envoy.admin.v3.SecretsConfigDump",
   "dynamic_active_secrets": [
    {
     "name": "default",
     "version_info": "2",
     "last_updated": "2022-10-15T10:35:38.847Z",
     "secret": {
      "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
      "name": "default",
      "tls_certificate": {
       "certificate_chain": {
        "inline_bytes": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUREakNDQWZhZ0F3SUJBZ0lRR3EwOWliZTVYNDl4T2VZRklVby81akFOQmdrcWhraUc5dzBCQVFzRkFEQWUKTVFzd0NRWURWUVFHRXdKVlV6RVBNQTBHQTFVRUNoTUdVMUJKUmtaRk1CNFhEVEl5TVRBeE5URXdNelV5T0ZvWApEVEl5TVRBeE5URXhNelV6T0Zvd1JURUxNQWtHQTFVRUJoTUNWVk14RGpBTUJnTlZCQW9UQlZOUVNWSkZNU1l3CkpBWURWUVFERXgxb1pXeHNiM2R2Y214a0xYWXlMVFUyTldRM1pHUTNaaTFqY0RnMmFEQlpNQk1HQnlxR1NNNDkKQWdFR0NDcUdTTTQ5QXdFSEEwSUFCRDMreXU0T25oTDVrZW4vVGR5K0FCdWhQRTJvSnVWeGUzWXJtWkhEOXdPQgpyd1BsQjhBNHE0bjhHR1NPVm9qUDJrY3Y1TEJrZGFjeS9JeHNNOUhmbHZLamdlc3dnZWd3RGdZRFZSMFBBUUgvCkJBUURBZ09vTUIwR0ExVWRKUVFXTUJRR0NDc0dBUVVGQndNQkJnZ3JCZ0VGQlFjREFqQU1CZ05WSFJNQkFmOEUKQWpBQU1CMEdBMVVkRGdRV0JCUnlFNVFuZkFGUWVmMlU4enRkVEhDSlVOc0lvREFmQmdOVkhTTUVHREFXZ0JRVgpzK0pMN2dCTzVoZW1GcU11c1c1bnJKUDhEakJwQmdOVkhSRUVZakJnZ2gxb1pXeHNiM2R2Y214a0xYWXlMVFUyCk5XUTNaR1EzWmkxamNEZzJhSUlWYUdWc2JHOTNiM0pzWkM1ellXMXdiR1V1YzNaamhpaHpjR2xtWm1VNkx5OW4KYjI5bmJHVXVZMjl0TDI1ekwzTmhiWEJzWlM5ellTOWtaV1poZFd4ME1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQgpBUUJrRDFnaHVvOFZhTUVBOXJMQ3A0VHM0cnV4QmV5K0FnWnBQMTRvM1I4RHhJSStHWFJDOStHS3RXOGNrQUxWClBrYXRMT1E3NEJCSXk0UHJiaGdQRkVVQ0wvaE9Kb24wazVmS29LbnA0OEFZZTQ2Tk5nckxzZEp3YVhzZzhmdVcKRGE2bExndlZQMkN6ckJxN3FHeHIxbjdJUytzQk1VL3dCaU9BbHlFb3lCTFJIbFg3N3I5dkpIOXNaVW9IckVaMApEMnBIVCtwb3RhNlVMcWQ5OHRIbzN4bnY2R0JxdnZuNXRCUTR3bWVSSnFKTlRTbWZTQ2ZrdmljTjE2VUN3ckNQCm9BeWhWYWJRRUVzcTd4bnFTclZqK2dmcXFQNzlSMldreERra0FOYS9CTUNna0RtM0p5ajJSYmFTYTZCWllhLzgKNnF6YW9wNlFFejJ1RlhoYkxPWW1wYlRoCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
       },
       "private_key": {
        "inline_bytes": "W3JlZGFjdGVkXQ=="
       }
      }
     }
    },
    {
     "name": "ROOTCA",
     "version_info": "1",
     "last_updated": "2022-10-15T10:35:38.910Z",
     "secret": {
      "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
      "name": "ROOTCA",
      "validation_context": {
       "trusted_ca": {},
       "custom_validator_config": {
        "name": "envoy.tls.cert_validator.spiffe",
        "typed_config": {
         "@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.SPIFFECertValidatorConfig",
         "trust_domains": [
          {
           "name": "aws.com",
           "trust_bundle": {
            "inline_bytes": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURKVENDQWcyZ0F3SUJBZ0lRQ1NqMGxNQ2IybVhnVUZOMDRiMFBWREFOQmdrcWhraUc5dzBCQVFzRkFEQWUKTVFzd0NRWURWUVFHRXdKVlV6RVBNQTBHQTFVRUNoTUdVMUJKUmtaRk1CNFhEVEl5TVRBeE5URXdNamt5TTFvWApEVEl5TVRBeE5qRXdNamt6TTFvd0hqRUxNQWtHQTFVRUJoTUNWVk14RHpBTkJnTlZCQW9UQmxOUVNVWkdSVENDCkFTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBSzZpUW5ucEsyQ3RXS3VrbTIvT0tJaDIKL1R2NGlTRGlWKzZROFZkWUJRRmQxcmVsUkEzaU0rRjRyQUgvZXFEOHFhSzJGajY5TkJTWGlWNytXdktvZHM4OQpHZW8xUGczV3o0cjl5aE56c0tXN0xuOGprbUFSMmY5RlgzUXAzR2h2QWZDR1V0aU02NkZHSXFsUUZNTmVWNVlCCm9DbzkwVG00TFpwYk1xTXFiZkkxcUlwYm5GUmFZclltU2t1NFB3ZzJVeWlxQUxvbUNEOThjcmQ3VUVUU1Rva1oKUFV5OCsrNXZwclBGSlhQQkg3VzlZdmRrelh5bUt4YXl5N1NPOUtma1JIc2NuWnNxLzhtb1owYndNaFhid2RjMwo2TjVhTFd2WXdDSytxL1ZoY0loRkh2MnJBUWYvcTNzVzh1OEkxdUZ2bzg5Mng3NVArNFpzdFBpMDZjT0Y0UXNDCkF3RUFBYU5mTUYwd0RnWURWUjBQQVFIL0JBUURBZ0VHTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME8KQkJZRUZEQXhST0R2bWNwQTA4UjVLV2RNRFQrczFWNFNNQnNHQTFVZEVRUVVNQktHRUhOd2FXWm1aVG92TDJGMwpjeTVqYjIwd0RRWUpLb1pJaHZjTkFRRUxCUUFEZ2dFQkFFYWVSRytoNG1hUEo5Vm9UanpYSE1nM1dzMVBKMVNSCkF4bzZ2YmszNUhxc0pRbUVnQ1hiZzlsODkvTXBJYUMySzNmQXJBRGdweEhCSU1oRWplZlhJZGFQRnloZDRjYmEKODg3QTF1V0RaOXhFUjNJU0ttek50Y09aRlA2S2dPWTlrMXpLNGp2cTJtdHRYdko2cTZlL1RXRDkrTjhBVS91Uwp1eGw3V3pkVGwrRUlNQ2FyNkRCMlVidUtLY1hQWnBUenVYMWtxSXV4N002Q3kzT1lOTlJ4MzhSKzZ2WWZrdGtuCjF4MlNINnc5blJMeGhlQ3BGeVBhWHc0RFFqOE55RGUvcWpaci9ReFhmRG4zQysyUms0alRQUFdZWTlscExxMGsKNVRxQXlndVpUZ2VLbDNJdXpMcXlBMlJ0Y3p0S3ZoOHF0UmFES0lleHZCWEFTQ054YWRVVUR0Zz0KLS0tLS1FTkQgQ0VSVElGSUNBVEUtLS0tLQo="
           }
          },
          {
           "name": "google.com",
           "trust_bundle": {
            "inline_bytes": "LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURLRENDQWhDZ0F3SUJBZ0lRUCtVVFh5SlhRVUFVMlhGU0dhSDBkekFOQmdrcWhraUc5dzBCQVFzRkFEQWUKTVFzd0NRWURWUVFHRXdKVlV6RVBNQTBHQTFVRUNoTUdVMUJKUmtaRk1CNFhEVEl5TVRBeE5URXdNamsxT1ZvWApEVEl5TVRBeE5qRXdNekF3T1Zvd0hqRUxNQWtHQTFVRUJoTUNWVk14RHpBTkJnTlZCQW9UQmxOUVNVWkdSVENDCkFTSXdEUVlKS29aSWh2Y05BUUVCQlFBRGdnRVBBRENDQVFvQ2dnRUJBTUc4UTBGYlhBUHBweW92UGNvVnlRRDMKdnh2OERpczlSQlc1WE9vSzRPd0FPMTlxYXFMbjAyWHRGUXBEblJ1bzhJVzhPLzJxbjU2Sytzc1BBS2JkTUtMbwpXYXJrZGdERVlTZFJzdTFlYmhlb2cxSmgzTVJ4YTJBUFJtVkpzeVJ6S1FxM284ZVBwM1ZPcEk5OGFSaE9LWXZCCmIwdERkTGtWU0NJaFJZZ0VaQXVoVHI1WlByZGdEYmJTSThtSmFnaTg2cldtR2xCYlR5VFh4ZTVkQitxQmtxc2kKUURpbHRCNU1xUTRBWEg4S2Mxbk5FZFlTL2RlSi9iVDZvM2cwZDhyUFI3bXJ4alVLN1Fhb1Q5eS8wQ1RjbU5ESQpFWVhMTGtGMHBtU3YwaWx6QjlUaU5NNDlyOER0QkxKdk1rWjg2ckpPWWEwQk1IMWJodDZTa1hob2RYRUpYRzhDCkF3RUFBYU5pTUdBd0RnWURWUjBQQVFIL0JBUURBZ0VHTUE4R0ExVWRFd0VCL3dRRk1BTUJBZjh3SFFZRFZSME8KQkJZRUZCV3o0a3Z1QUU3bUY2WVdveTZ4Ym1lc2svd09NQjRHQTFVZEVRUVhNQldHRTNOd2FXWm1aVG92TDJkdgpiMmRzWlM1amIyMHdEUVlKS29aSWh2Y05BUUVMQlFBRGdnRUJBS2UxT21LQ1pGejE3RklrYkROTUlxVExtNTRRCmt2VlRSZXdSR0FYWUQ0aXJlcDltR01PMjBXeTUxQ1cxSmFyN1JrenV0WENNQjZGYy9WYXdMb3k4bE9CL0xNc2wKRjlicGRJKzgzUmZrcWVsY1hvRWt3dEhvTXdBZHE0RG9yZVpKUVVRbGlZaDQvTDlHUkVIa1JyTDV1WWhmSURsaQpLK3pkenVRYkRzbzlZc1M0SVByYkc3QndsWmpaYVRBT1lmT0V6N0oxeTk2YWp4ZzhZeFdKSG1Kb3ViTTBtclRrCnJtN0lUbnBSZ1h1U3NySy9LSFpqMHdiNTgza1prQVVxL0lldFdiQkt0a3pITVNSK2dDT0FkbUR6b0wralVlWXkKUDFkelhqdmdMQ0RDVWJZSzArUG8zcWtjaTZGYTlKNGxQR2drc1djSFRrTkpuajlqWGU2L1FNbk5SdUE9Ci0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"
           }
          }
         ]
        }
       }
      }
     }
    }
   ]
  }
 ]
}

attaching the entire envoy config of helloworld app

helloworld-v2-config-dump.json.zip

Enhance AWS node attestor server plugin to validate EC2 instances across multiple AWS account.

We are trying to set up the spire infrastructure in our production environment. The production environment has multiple AWS accounts. We want to deploy a spire server in one of the accounts and attest the EC2 hosts in all other AWS accounts using AWS aws_iid node attestor.

When both the spire server and the spire agent runs in the same AWS account, the spire server is able to attest to the EC2 host. However when spire server and spire agent are in different AWS account, the node attestation fails with this message: ERRO[0000] agent crashed error=“failed to get SVID: error getting attestation response from SPIRE server: rpc error: code = Internal desc = failed to attest: rpc error: code = Unknown desc = aws-iid: attempted attestation but an error occurred querying AWS via describe-instances: InvalidInstanceID.NotFound:
SPIRE Agent Windows Support
The SPIRE Agent should be extended to support attestation of Windows workloads. The current version of Windows supports UDS natively as of Windows 10 as of build 1809. @azdagron Has done a POC using named pipes.

What are the introspection capabilities through named pipes/windows UDS?

How much does support of Windows users before build 1809 matter?

User experience for OS-specific plugins, i.e. UNIX workload attestor.

Version: n/a

Platform: Windows

Subsystem: Agent
Implement upstream authority plugin for GCP
Signed-off-by: ramand.dragcp [email protected]

Pull Request check list

[*] Commit conforms to CONTRIBUTING.md?

[*] Proper tests/regressions included?

[*] Documentation updated?

Affected functionality Adds a new plugin for GCP Certificate Authority Service backed upstream authority

Description of change Thank you for reviewing the PR #2039 by @Jonpez2. I am continuing that PR here. I believe I have incorporated all of the feedback

A few notes:

Changes to go.sum and go.mod happened automatically as part of "make build"

We pick all the CAs in GCP CAS that are in enabled state and also matches the specified label key/value pair. The one with the earliest expiration is used to create and sign intermediate CAs

I have tried to included relevant documentation links for most of the APIs and types of GCP CAS.

Which issue this PR fixes
Safer DB migration strategy in spire server cluster env
Currently if SPIRE Server with DB version X starts up and looks to DB with version < X, SPIRE will auto-migrate the DB to X. Backwards compatibility does not exist.

Problems:

In a distributed environment this is fatal to the other SPIRE Servers. They will be in a < X DB version and fail to start due to not being able to migrate backwards

The migration is uncontrolled; in such an environment we not commit to every single Server must upgrade, and hope that the new DB schema is good

Some (non-mutually exclusive) Solutions:

A flag to pass to Server config to not perform auto-migration; without other changes this is fatal to the newer server but lets the older ones continue operations

Proper semantic versioning of the DB versions; when starting up and the DB is at a later version than the Server expects, if the new version does not indicate a breaking change then the old Server should be able to continue operations (not perform backwards migration)

A method of rolling back the DB version

A method of a later Server being backwards compatible for older DB versions
Implement an Informer strategy for k8s-workload-registrar

The registrar gains an option to use an informer instead of a webhook. In this mode, it watches the k8s API instead of listening for updates from a webhook.

The controller code is extended so that when entries are added, any outdated entries for the same pod are removed. This means that label changes are now reflected in the registration entries.

Update client-go version to kubernetes 1.15 to get past API changes. Regenerate mocks to match.
Optimize bundle conversion amount in spire server TLS connections
Version: Any version

Platform: Any Platform

Subsystem: Server

After https://github.com/spiffe/spire/pull/3642 is merged, spire server will perform a bundle conversion ( *common.bundle to *x509bundle.Bundle) in all tls connections. In the current implementation, we are using a cache for the bundles fetched from the datastore. We could somehow include the parsed version of the bundle in this cache to optimize the conversion amount.

My initial thoughts on this are about not changing the DataStore interface to maintain the scope of the changes to a minimum. I think we can achieve that by:

Adding the parsed bundle version to the cache entry in DatastoreCache

type bundleEntry struct { mu sync.Mutex ts time.Time bundle *common.Bundle x509Bundle *x509bundle.Bundle }

Create a new method in DatastoreCache for retrieving this parsed data

func (ds *DatastoreCache) FetchBundleX509(ctx context.Context, trustDomain string) (*x509bundle.Bundle, error) {

Change the Endpoints struct to use the concrete DatastoreCache type instead of the Datastore interface; we are already using the DatastoreCache here by default, so this shouldn't break any existing integration test.

I am willing to open a PR on this in case of approval, and I would like to get some opinions here. What do you think?
Bump github.com/aws/aws-sdk-go-v2/service/secretsmanager from 1.17.0 to 1.18.0
Bumps github.com/aws/aws-sdk-go-v2/service/secretsmanager from 1.17.0 to 1.18.0.

Changelog

Sourced from github.com/aws/aws-sdk-go-v2/service/secretsmanager's changelog.

Release (2023-01-05)

General Highlights

Dependency Update: Updated to the latest SDK module versions

Module Highlights

github.com/aws/aws-sdk-go-v2/service/accessanalyzer: v1.19.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/account: v1.8.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/acm: v1.17.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/acmpca: v1.20.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/alexaforbusiness: v1.15.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/amp: v1.16.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/amplify: v1.13.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/amplifybackend: v1.14.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

Feature: Updated GetBackendAPIModels response to include ModelIntrospectionSchema json string

github.com/aws/aws-sdk-go-v2/service/amplifyuibuilder: v1.9.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/apigateway: v1.16.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/apigatewaymanagementapi: v1.11.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/apigatewayv2: v1.13.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/appconfig: v1.15.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/appconfigdata: v1.5.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/appflow: v1.23.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/appintegrations: v1.14.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/applicationautoscaling: v1.17.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/applicationcostprofiler: v1.10.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/applicationdiscoveryservice: v1.15.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/applicationinsights: v1.17.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/appmesh: v1.17.0

Feature: Add ErrorCodeOverrideaws/smithy-go#401

github.com/aws/aws-sdk-go-v2/service/apprunner: v1.16.0

... (truncated)

Commits

8d0db3b Release 2022-11-11

409cb2a Regenerated Clients

1ef6c0c Update endpoints model

fb60a6e Update API model

3d79564 Add token refresh support to SSOCredentialProvider (#1903)

0966539 Release 2022-11-10

aec7ab6 Regenerated Clients

d008171 Update endpoints model

29d44eb Update API model

01cee3e Release 2022-11-09

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
Feature Request: Support Dynamic pkix Subject
according to https://github.com/spiffe/spire/blob/v1.5.3/doc/spire_server.md#server-configuration-file, we can update ca_subject in spire-server, cloud we update pki Subject info in spire-server? such as Country or Organization?

&x509.Certificate{ Subject: pkix.Name{ Country: []string{"US"}, Organization: []string{"SPIRE"}, },

see https://github.com/spiffe/spire/blob/main/cmd/spire-server/cli/run/run.go#L51
Spire Plugin supports more authentication options besides kubeconfg
According to https://github.com/spiffe/spire/blob/v1.5.3/doc/plugin_server_upstreamauthority_cert_manager.md document,

UpstreamAuthority "cert-manager" { plugin_data { issuer_name = "spire-ca" issuer_kind = "Issuer" issuer_group = "cert-manager.io" namespace = "sandbox" kube_config_file = "/etc/kubernetes/kubeconfig" } }

The kube_config_file parameter is provided, whether spire-server cert-manager can support other options.

Version: 1.5.3

Platform: linux

Subsystem: server:1.5.3 agent:1.5.3
Run integration tests for arm64 on release builds

#3607 introduces support for arm64-based release container images. Integration tests should be run against these artifacts in the release pipeline, at a minimum.

SPIRE is a toolchain of APIs for establishing trust between software systems across a wide variety of hosting platforms

Get SPIRE

Learn about SPIRE

Integrate with SPIRE

Contribute to SPIRE

Further Reading

Security

Security Assessments

Reporting Security Vulnerabilities

Owner

SPIFFE

Comments

Add Spiffe ID templating for k8s-workload-registrar

[RFC] Serverless architecture support

[RFC] Serverless architecture support

Background

Proposal

Sample implementation

SPIRE

Serverless environment

Request for Comments

Propose Updated Protos for the SPIRE Server API

spire-server too high CPU usage

Failed to collect all selectors for PID" error="workload attestor \"k8s\" failed: rpc error: code = DeadlineExceeded desc = workloadattestor(k8s): no selectors found after max poll attempts"

Spire do not federate between k8s clusters which implements Istio multi-primary on different networks

Enhance AWS node attestor server plugin to validate EC2 instances across multiple AWS account.

SPIRE Agent Windows Support

Implement upstream authority plugin for GCP

Safer DB migration strategy in spire server cluster env

Implement an Informer strategy for k8s-workload-registrar

Optimize bundle conversion amount in spire server TLS connections

Bump github.com/aws/aws-sdk-go-v2/service/secretsmanager from 1.17.0 to 1.18.0

Release (2023-01-05)

General Highlights

Module Highlights

Feature Request: Support Dynamic pkix Subject

Spire Plugin supports more authentication options besides kubeconfg

Run integration tests for arm64 on release builds

Related tags

Scans and catches callbacks of systems that are impacted by Log4J Log4Shell vulnerability across specific headers.

Obfuscate Go code by wrapping the Go toolchain

Secret management toolchain

zero-trust remote firewall instrumentation

Update-java-ca-certificates - Small utility to convert the system trust store to a system Java KeyStore

AI-Powered Code Reviews for Best Practices & Security Issues Across Languages

Find secrets and passwords in container images and file systems

Scanner to send specially crafted requests and catch callbacks of systems that are impacted by Log4J Log4Shell vulnerability (CVE-2021-44228)

Volana - Shell command obfuscation to avoid detection systems

Scan systems and docker images for potential spring4shell vulnerabilities.

Implementations of the Coconut signing scheme, cross-compatible between Rust and Go.

Static binary analysis tool to compute shared strings references between binaries and output in JSON, YAML and YARA

Gorsair hacks its way into remote docker containers that expose their APIs

Secure software enclave for storage of sensitive information in memory.

A software supply chain security inspection tool.

Application - A software-as-a-service available to clients globally as a multi-tenant elastic solution based on the Bhojpur.NET Platform

This is a "simple" game server. Main functionalities are matching and establishing a connection between players

Use AWS SQS as a clipboard to copy and paste across different systems and platforms

Easy to use cryptographic framework for data protection: secure messaging with forward secrecy and secure data storage. Has unified APIs across 14 platforms.

Easy to use cryptographic framework for data protection: secure messaging with forward secrecy and secure data storage. Has unified APIs across 14 platforms.