Use eBPF to speed up your Service Mesh like crossing an Einstein-Rosen Bridge.

merbridge

Use eBPF to speed up your Service Mesh like crossing an Einstein-Rosen Bridge.

Usage

Install

You just only need to run the following command to your Istio cluster to get eBPF to speed up Istio:

kubectl apply -f https://raw.githubusercontent.com/merbridge/merbridge/main/deploy/all-in-one.yaml

Or on Linkerd cluster:

kubectl apply -f https://raw.githubusercontent.com/merbridge/merbridge/main/deploy/all-in-one-linkerd.yaml

Note: currently only works on Linux kernel >= 5.7, run uname -r check your kernel version before install merbridge.

If you want to install Merbridge by Helm, read the guidelines: Deploy Merbridge with Helm.

Uninstall

  • Istio:
kubectl delete -f https://raw.githubusercontent.com/merbridge/merbridge/main/deploy/all-in-one.yaml
  • Linkerd:
kubectl delete -f https://raw.githubusercontent.com/merbridge/merbridge/main/deploy/all-in-one-linkerd.yaml

Get involved

Join the Merbridge slack.

Contributors

Made with contrib.rocks.

License

Copyright 2022 the Merbridge Authors. All rights reserved.

Licensed under the Apache License, Version 2.0.

Landscapes

  

Merbridge enriches the CNCF CLOUD NATIVE Landscape.

Comments
  • ADD cniserver listen fd backup

    ADD cniserver listen fd backup

    1.Provides a method to save cniserver listen FDs when the cniserver container is restarted or pod is recreate. 2.The general idea is to do a temporary save via a FDs-back server roll-in roll-out. 3.Currently the code is simply implemented and needs further optimization and testing. We look forward to community discussion and further guidance on optimization.

  • Why don't print redirect xxx bytes with eBPF successfully? It means redirect failed?

    Why don't print redirect xxx bytes with eBPF successfully? It means redirect failed?

    I tested this project in my local machine, and i found when it exec redirect(mb_redir.c), it never print debugf("redirect %d bytes with eBPF successfully", msg->size);, so it means redirect failed? In this case, the bookinfo example runs successfully.

    __section("sk_msg") int mb_msg_redir(struct sk_msg_md *msg)
    {
        struct pair p = {
            .dip = msg->local_ip4,
            .dport = bpf_htons(msg->local_port),
            .sip = msg->remote_ip4,
            .sport = msg->remote_port >> 16,
        };
        long ret = bpf_msg_redirect_hash(msg, &sock_pair_map, &p, BPF_F_INGRESS);
        if (ret)
            debugf("redirect %d bytes with eBPF successfully", msg->size);
        return 1;
    }
    

    why the key is :

        struct pair p = {
            .dip = msg->local_ip4,
            .dport = bpf_htons(msg->local_port),
            .sip = msg->remote_ip4,
            .sport = msg->remote_port >> 16,
        };
    

    i change the key like this:

        struct pair p = {
            .sip = msg->local_ip4,
            .sport = bpf_htons(msg->local_port),
            .dip = msg->remote_ip4,
            .dport = msg->remote_port >> 16,
        };
    

    and it print the log like:

    python-933173  [005] d... 17818.380818: bpf_trace_printk: [debug] redirect 314 bytes with eBPF successfully
    

    but the bookinfo example i tested is failed because the connection was closed, it can't connect to details and reviews. It confuses me.

  • `failed to pin program cgroup/connect4` in `kind`

    `failed to pin program cgroup/connect4` in `kind`

    $ uname -r
    5.10.46-5rodete1-amd64
    

    Installed with

    kubectl apply -f https://raw.githubusercontent.com/merbridge/merbridge/main/deploy/all-in-one.yaml
    

    Kubernetes:

    Server Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.0", GitCommit:"ab69524f795c42094a6630298ff53f3c3ebab7f4", GitTreeState:"clean", BuildDate:"2021-12-07T21:30:26Z", GoVersion:"go1.17.3", Compiler:"gc", Platform:"linux/amd64"}
    

    running in kind

  • use cgroup_ips map to improve mb_connect performance

    use cgroup_ips map to improve mb_connect performance

    bpf_sk_lookup_tcp may cost 50-150ns per ops, It is more expensive.

    Without cache:

     kebe@Kebe-DaoCloud-PC  ~/workspace/mebpf   main ±  sudo go test -bench .
    goos: linux
    goarch: amd64
    pkg: github.com/merbridge/merbridge
    cpu: AMD Ryzen 9 3900 12-Core Processor             
    BenchmarkConnect/eBPF-24                1000000000               0.4532 ns/op
    BenchmarkConnect/eBPF/mb_sock_connect        108              3203 ns/op
    BenchmarkSockops/eBPF-24                1000000000               0.4327 ns/op
    BenchmarkSockops/eBPF/mb_sockops             954               689.2 ns/op
    PASS
    ok      github.com/merbridge/merbridge  20.530s
    

    After cached:

     kebe@Kebe-DaoCloud-PC  ~/workspace/mebpf   main ±  sudo go test -bench .
    goos: linux
    goarch: amd64
    pkg: github.com/merbridge/merbridge
    cpu: AMD Ryzen 9 3900 12-Core Processor             
    BenchmarkConnect/eBPF-24                1000000000               0.4539 ns/op
    BenchmarkConnect/eBPF/mb_sock_connect        107              2475 ns/op
    BenchmarkSockops/eBPF-24                1000000000               0.4545 ns/op
    BenchmarkSockops/eBPF/mb_sockops             963               672.1 ns/op
    PASS
    ok      github.com/merbridge/merbridge  21.205s
    

    The optimization results in a 20% performance improvement for connect programs.

  • feat: add support for kuma service mesh

    feat: add support for kuma service mesh

    • Updated make files to build helm chart and k8s deployment manifest with Kuma support
    • Bumped helm chart to 0.5.0
    • Kuma supports DNS redirection like Istio does, so I updated ebpf files which so far were only used for Istio, to also be compiled for Kuma
    • As Kuma also supports dynamic reload, I updated hack/reload-prog.sh to be able to use it with Kuma
      • shebang has to be in the first line of the script, so I moved it there
      • as the shell code is definitelly written with bash in mind (i.e. usage of arrays), I changed shebang from sh to bash
    • Updated all README.md files with kuma-related instructions and informations
    • Added scripts/install-kuma.sh script which can be used to install kuma and wait till the deployment will be ready
    • Modified github actions workflow to include e2e test for kuma

    Signed-off-by: Bart Smykla [email protected]

  • Add support for ipv6

    Add support for ipv6

    To enable ipv6 only, just add the following env in the daemon set:

    env:
      - name: ENABLE_IPV4
        value: "false"
      - name: ENABLE_IPV6
        value: "true"
    

    Note that ipv6 requires cni-mode enabled

  • Merbridge doesn't work on EKS

    Merbridge doesn't work on EKS

    Bug Description

    Merbridge fails to start on EKS.

    OS: linux (arm64) OS Image: Bottlerocket OS 1.11.1 (aws-k8s-1.24) Kernel version: 5.15.59

    logs

    2022-12-09T14:45:58.254315Z	warn	OS CA Cert could not be found for agent
    Error: failed to load ebpf programs: unexpected exit code: -1, err: fork/exec /usr/bin/make: exec format error
    Usage:
      mbctl [flags]
    
    Flags:
          --cni-bin-dir string      /opt/cni/bin mount path (default "/host/opt/cni/bin")
          --cni-config-dir string   /etc/cni/net.d mount path (default "/host/etc/cni/net.d")
          --cni-mode                Enable Merbridge CNI plugin
      -d, --debug                   Debug mode
          --enable-hot-restart      enable hot restart
      -h, --help                    help for mbctl
          --host-proc string        /proc mount path (default "/host/proc")
          --host-var-run string     /var/run mount path (default "/host/var/run")
      -k, --kind                    Enable when Kubernetes is running in Kind
          --kubeconfig string       Kubernetes configuration file
          --kubecontext string      The name of the kube config context to use
      -m, --mode string             Service mesh mode, current support istio, linkerd and kuma (default "istio")
      -r, --use-reconnect           Use re-connect mode for same-node acceleration (default true)
    2022-12-09T14:45:58.256686279Z
    

    helm

    values: {
      namespace: 'linkerd',
      mode: 'linkerd',
      cniMode: true,
      image: {
        tag: '0.7.2',
        pullPolicy: 'IfNotPresent',
      },
      resources: {
        container: {
          limit: { memory: '200Mi' },
          request: {
            cpu: '100m',
            memory: '200Mi',
          },
        },
      },
    },
    

    Version

    OS: `linux (arm64)`
    OS Image: `Bottlerocket OS 1.11.1 (aws-k8s-1.24)`
    Kernel version: `5.15.59`
    
  • Bump github.com/spf13/cobra from 1.5.0 to 1.6.0

    Bump github.com/spf13/cobra from 1.5.0 to 1.6.0

    Bumps github.com/spf13/cobra from 1.5.0 to 1.6.0.

    Release notes

    Sourced from github.com/spf13/cobra's releases.

    v1.6.0

    Summer 2022 Release

    Some exciting changes make their way to Cobra! Command completions continue to get better and better (including adding --help and --version automatic flags to the completions list). Grouping is now possible in your help output as well! And you can now use the OnFinalize method to cleanup things when all "work" is done. Checkout the full changelog below:


    Features 🌠

    Deprecation 👎🏼

    • ExactValidArgs is deprecated (but not being removed entirely). This is abit nuanced, so checkout #1643 for further information and the updated user_guide.md on how this may affect you (and how you can take advantage of the correct behavior in the validators): @​umarcor #1643

    Bug fixes 🐛

    Dependencies 🗳️

    Testing 🤔

    Docs ✏️

    Misc 💭

    Note: Per #1804, we will be moving away from "seasonal" releases and doing more generic point release targets. Continue to track the milestones and issues in the spf13/cobra GitHub repository for more information!

    Great work everyone! Cobra would never be possible without your contributions! 🐍

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Bump k8s.io/client-go from 0.23.6 to 0.24.0

    Bump k8s.io/client-go from 0.23.6 to 0.24.0

    Bumps k8s.io/client-go from 0.23.6 to 0.24.0.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • [WIP]add /approve support for ci-bot merge

    [WIP]add /approve support for ci-bot merge

    Fixes #67

    Comment /merge in later PR, it will be merged.

    You have to manage the APPROVERS list here. (The current version of gh-ci-bot cannot read from https://github.com/merbridge/merbridge/blob/059c65629628b6b4df73dc2e02038cc1c7b9ef57/CODEOWNERS 😄)

  • Bump github.com/cilium/ebpf from 0.8.1 to 0.9.0

    Bump github.com/cilium/ebpf from 0.8.1 to 0.9.0

    Bumps github.com/cilium/ebpf from 0.8.1 to 0.9.0.

    Release notes

    Sourced from github.com/cilium/ebpf's releases.

    Releasing the BTF package!

    This release makes package btf public, allowing type information embedded in eBPF ELF objects and vmlinux to be read programmatically from Go. This enables use cases like searching and inspecting the kernel's types and function signatures at runtime, or ensuring alignment between structures defined in Go and C. Package btf stands on its own and does not require adopting other subpackages, potentially making it useful beyond the domain of eBPF. Additionally, parsing BTF was made significantly faster and now consumes less memory.

    Note: Modifications to btf.Types are not (yet) reflected in the BTF info loaded into the kernel. This will be implemented in the near future, and will provide the flexibility of, for example, creating maps with arbitrarily crafted key/value type information for pretty-printing map dumps, among many other exciting use cases.

    Users of the ringbuf and perf packages can now avoid allocations when reading samples with the addition of ReadInto().

    Feature probes for program helpers (features.HaveProgramHelper()) were added, as well as a few miscellaneous probes for large instruction limit, bounded loops and the supported eBPF ISA. This now brings the library mostly on par with bpftool's probing capabilities, except for a few program types that require BTF.

    There was also one important bugfix where CO-RE relocations in bpf2bpf subprograms were not applied. Users are strongly encouraged to upgrade.

    Enjoy!

    Breaking changes

    • Deprecations:
      • CollectionSpec.RewriteMaps is deprecated in favour of CollectionOptions.MapReplacements
      • features.HaveProgType is deprecated in favour of features.HaveProgramType
    • CollectionSpec.BTF is now called CollectionSpec.Types
    • link.K(ret)probe and link.Tracepoint signatures have changed, they now take an additional options parameter. Pass nil to retain the old behaviour.

    Features

    Bug fixes

    Miscellaneous

    ... (truncated)

    Commits
    • 951bb28 features: rename HaveProgType API
    • d1edf5a features: add HaveProgramHelper API
    • c4f6259 btf: Add spec types iterator
    • 00ae3f2 testdata: loader - declare constant in custom .rodata.test section
    • 1ff15d2 collection: make RewriteConstants operate on custom .rodata* sections
    • 20cccef testdata: loader - read from anonymous (global) constant
    • 9850db7 elf_reader: tolerate untyped/local map relocations from llvm 7/9
    • 1e37e4f elf_reader: accept ELF data sections without a corresponding BTF datasec
    • 88c2d0c elf_reader: freeze all data sections with .rodata* prefix
    • 1e1f08c Makefile: disable journald logging in podman
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Bump k8s.io/client-go from 0.25.1 to 0.26.0

    Bump k8s.io/client-go from 0.25.1 to 0.26.0

    Bumps k8s.io/client-go from 0.25.1 to 0.26.0.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Bump k8s.io/apimachinery from 0.25.1 to 0.26.0

    Bump k8s.io/apimachinery from 0.25.1 to 0.26.0

    Bumps k8s.io/apimachinery from 0.25.1 to 0.26.0.

    Commits
    • 5d4cdd2 Merge remote-tracking branch 'origin/master' into release-1.26
    • 6cbc4a3 Update golang.org/x/net 1e63c2f
    • 6561235 Merge pull request #113699 from liggitt/manjusaka/fix-107415
    • dad8cd8 Update workload selector validation
    • fe82462 Add extra value validation for matchExpression field in LabelSelector
    • 067949d update k8s.io/utils to fix util tracing panic
    • 0ceff90 Merge pull request #112223 from astraw99/fix-ownerRef-validate
    • 9e85d3a Merge pull request #112649 from howardjohn/set/optimize-everything-nothing
    • 88a1448 Rename and comment on why sharing is safe
    • b03a432 Merge pull request #113367 from pohly/dep-ginkgo-gomega
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Bump k8s.io/api from 0.25.1 to 0.26.0

    Bump k8s.io/api from 0.25.1 to 0.26.0

    Bumps k8s.io/api from 0.25.1 to 0.26.0.

    Commits
    • 2ee9a6c Update dependencies to v0.26.0 tag
    • 07ac8fe Merge remote-tracking branch 'origin/master' into release-1.26
    • 566ee01 Update golang.org/x/net 1e63c2f
    • b966dc9 sync: update go.mod
    • 053624e Merge pull request #111023 from pohly/dynamic-resource-allocation
    • 3590eda Merge pull request #113375 from atiratree/PodHealthyPolicy-api
    • 5a4f9a5 generated
    • 5cb3202 Merge pull request #113186 from ttakahashi21/KEP-3294
    • 993c43c api: add UnhealthyPodEvictionPolicy for PDBs
    • dfd6ea2 Generate code
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Bump golang from 1.19.2 to 1.19.4

    Bump golang from 1.19.2 to 1.19.4

    Bumps golang from 1.19.2 to 1.19.4.

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Random `Failed to create pod sandbox` errors for all pods on nodes

    Random `Failed to create pod sandbox` errors for all pods on nodes

    Bug Description

    Pods are failing to start with the following error on certain nodes. I could not find any obvious patterns of why pods on some nodes work fine and fail on others. These symptoms may indicate some racing issues.

    Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "3c0ce05181c3c2383ac85856ac0c0ad5bbab69fd16e123eb0268dc602b28c28b": failed to find network info for sandbox "3c0ce05181c3c2383ac85856ac0c0ad5bbab69fd16e123eb0268dc602b28c28b"
    

    Helm values

    namespace: linkerd
    mode: linkerd
    cniMode: true
    resources:
      container:
        limit:
          memory: 200Mi
        request:
          cpu: 100m
          memory: 200Mi
    

    1st failing node details OS: linux (arm64) OS Image: Bottlerocket OS 1.11.1 (aws-k8s-1.24) Kernel version: 5.15.59 Container runtime: containerd://1.6.8+bottlerocket Kubelet version: v1.24.6-eks-4360b32 AWS EC2 instance type: t4g.small

    ** Merbridge logs

    2022-12-10T17:21:55.083303Z	warn	OS CA Cert could not be found for agent
    [ -f bpf/mb_connect.c ] && make -C bpf load || make -C bpf load-from-obj
    make[1]: Entering directory '/app/bpf'
    clang -O2 -g  -Wall -target bpf -I/usr/include/aarch64-linux-gnu  -DMESH=2 -DENABLE_IPV4=1 -DENABLE_IPV6=0 -c mb_connect.c -o mb_connect.o
    clang -O2 -g  -Wall -target bpf -I/usr/include/aarch64-linux-gnu  -DMESH=2 -DENABLE_IPV4=1 -DENABLE_IPV6=0 -c mb_get_sockopts.c -o mb_get_sockopts.o
    clang -O2 -g  -Wall -target bpf -I/usr/include/aarch64-linux-gnu  -DMESH=2 -DENABLE_IPV4=1 -DENABLE_IPV6=0 -c mb_redir.c -o mb_redir.o
    clang -O2 -g  -Wall -target bpf -I/usr/include/aarch64-linux-gnu  -DMESH=2 -DENABLE_IPV4=1 -DENABLE_IPV6=0 -c mb_sockops.c -o mb_sockops.o
    clang -O2 -g  -Wall -target bpf -I/usr/include/aarch64-linux-gnu  -DMESH=2 -DENABLE_IPV4=1 -DENABLE_IPV6=0 -c mb_bind.c -o mb_bind.o
    clang -O2 -g  -Wall -target bpf -I/usr/include/aarch64-linux-gnu  -DMESH=2 -DENABLE_IPV4=1 -DENABLE_IPV6=0 -c mb_sendmsg.c -o mb_sendmsg.o
    clang -O2 -g  -Wall -target bpf -I/usr/include/aarch64-linux-gnu  -DMESH=2 -DENABLE_IPV4=1 -DENABLE_IPV6=0 -c mb_recvmsg.c -o mb_recvmsg.o
    clang -O2 -g  -Wall -target bpf -I/usr/include/aarch64-linux-gnu  -DMESH=2 -DENABLE_IPV4=1 -DENABLE_IPV6=0 -c mb_tc.c -o mb_tc.o
    sudo mount -t bpf bpf /sys/fs/bpf
    sudo mkdir -p /sys/fs/bpf/tc/globals
    [ -f /sys/fs/bpf/cookie_original_dst ] || sudo bpftool map create /sys/fs/bpf/cookie_original_dst type lru_hash key 8 value 24 entries 65535 name cookie_original_dst
    [ -f /sys/fs/bpf/local_pod_ips ] || sudo bpftool map create /sys/fs/bpf/local_pod_ips type hash key 16 value 244 entries 1024 name local_pod_ips
    [ -f /sys/fs/bpf/process_ip ] || sudo bpftool map create /sys/fs/bpf/process_ip type lru_hash key 4 value 4 entries 1024 name process_ip
    [ -f /sys/fs/bpf/cgroup_info_map ] || sudo bpftool map create /sys/fs/bpf/cgroup_info_map type lru_hash key 8 value 32 entries 1024 name cgroup_info_map
    [ -f /sys/fs/bpf/mark_pod_ips_map ] || sudo bpftool map create /sys/fs/bpf/mark_pod_ips_map type hash key 4 value 16 entries 65535 name mark_pod_ips_map
    sudo bpftool -m prog loadall mb_connect.o /sys/fs/bpf/connect \
    	map name cookie_original_dst pinned /sys/fs/bpf/cookie_original_dst \
    	map name local_pod_ips pinned /sys/fs/bpf/local_pod_ips \
    	map name mark_pod_ips_map pinned /sys/fs/bpf/mark_pod_ips_map \
    	map name process_ip pinned /sys/fs/bpf/process_ip \
    	map name cgroup_info_map pinned /sys/fs/bpf/cgroup_info_map
    [ -f /sys/fs/bpf/pair_original_dst ] || sudo bpftool map create /sys/fs/bpf/pair_original_dst type lru_hash key 36 value 24 entries 65535 name pair_original_dst
    [ -f /sys/fs/bpf/sock_pair_map ] || sudo bpftool map create /sys/fs/bpf/sock_pair_map type sockhash key 36 value 4 entries 65535 name sock_pair_map
    sudo bpftool -m prog load mb_sockops.o /sys/fs/bpf/sockops \
    	map name cookie_original_dst pinned /sys/fs/bpf/cookie_original_dst \
    	map name process_ip pinned /sys/fs/bpf/process_ip \
    	map name pair_original_dst pinned /sys/fs/bpf/pair_original_dst \
    	map name sock_pair_map pinned /sys/fs/bpf/sock_pair_map
    sudo bpftool -m prog load mb_get_sockopts.o /sys/fs/bpf/get_sockopts \
    	map name pair_original_dst pinned /sys/fs/bpf/pair_original_dst
    sudo bpftool -m prog load mb_redir.o /sys/fs/bpf/redir \
    	map name sock_pair_map pinned /sys/fs/bpf/sock_pair_map
    sudo bpftool -m prog load mb_bind.o /sys/fs/bpf/bind
    sudo bpftool -m prog loadall mb_sendmsg.o /sys/fs/bpf/sendmsg \
    	map name cookie_original_dst pinned /sys/fs/bpf/cookie_original_dst \
    	map name mark_pod_ips_map pinned /sys/fs/bpf/mark_pod_ips_map \
    	map name cgroup_info_map pinned /sys/fs/bpf/cgroup_info_map
    sudo bpftool -m prog loadall mb_recvmsg.o /sys/fs/bpf/recvmsg \
    	map name cookie_original_dst pinned /sys/fs/bpf/cookie_original_dst \
    	map name mark_pod_ips_map pinned /sys/fs/bpf/mark_pod_ips_map \
    	map name cgroup_info_map pinned /sys/fs/bpf/cgroup_info_map
    make[1]: Leaving directory '/app/bpf'
    time="2022-12-10T17:21:57Z" level=info msg="Copied /app/merbridge-cni to /host/opt/cni/bin." func="cni-server.copyBinaries()" file="install.go:363"
    time="2022-12-10T17:21:57Z" level=info msg="write kubeconfig file /host/etc/cni/net.d/ZZZ-merbridge-cni-kubeconfig with: \n# Kubeconfig file for Merbridge CNI plugin.\napiVersion: v1\nkind: Config\nclusters:\n- name: local\n  cluster:\n    server: https://[10.100.0.1]:443\n    insecure-skip-tls-verify: true\nusers:\n- name: merbridge-cni\n  user:\n    token: \"<redacted>\"\ncontexts:\n- name: merbridge-cni-context\n  context:\n    cluster: local\n    user: merbridge-cni\ncurrent-context: merbridge-cni-context\n" func="cni-server.createKubeconfigFile()" file="install.go:453"
    time="2022-12-10T17:21:57Z" level=info msg="CNI config file /host/etc/cni/net.d/01-linkerd-cni.conf exists. Proceeding." func="cni-server.getCNIConfigFilepath()" file="install.go:310"
    time="2022-12-10T17:21:57Z" level=info msg="Renaming /host/etc/cni/net.d/01-linkerd-cni.conf extension to .conflist" func="cni-server.writeCNIConfig()" file="install.go:259"
    time="2022-12-10T17:21:57Z" level=info msg="Created CNI config /host/etc/cni/net.d/01-linkerd-cni.conflist" func="cni-server.writeCNIConfig()" file="install.go:267"
    time="2022-12-10T17:21:57Z" level=info msg="Pod Watcher Ready" func="controller.RunLocalPodController()" file="pod.go:53"
    make -C bpf attach
    make[1]: Entering directory '/app/bpf'
    sudo bpftool cgroup attach /sys/fs/cgroup/unified connect4 pinned /sys/fs/bpf/connect/cgroup_connect4
    sudo bpftool cgroup attach /sys/fs/cgroup/unified sock_ops pinned /sys/fs/bpf/sockops
    sudo bpftool cgroup attach /sys/fs/cgroup/unified getsockopt pinned /sys/fs/bpf/get_sockopts
    sudo bpftool prog attach pinned /sys/fs/bpf/redir msg_verdict pinned /sys/fs/bpf/sock_pair_map
    sudo bpftool cgroup attach /sys/fs/cgroup/unified bind4 pinned /sys/fs/bpf/bind
    sudo bpftool cgroup attach /sys/fs/cgroup/unified sendmsg4 pinned /sys/fs/bpf/sendmsg/cgroup_sendmsg4
    sudo bpftool cgroup attach /sys/fs/cgroup/unified recvmsg4 pinned /sys/fs/bpf/recvmsg/cgroup_recvmsg4
    make[1]: Leaving directory '/app/bpf'
    time="2022-12-10T17:22:03Z" level=info msg="cni called delete with args: {ContainerID:ce89bae7ebf0b4a62c08aad3db9eb1a913d490b4eea208dd5019ffe4522cdbc2 Netns:/var/run/netns/cni-417e5717-d563-9332-ae6e-1e5c267177f1 IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=ebs-csi-node-qpr8p;K8S_POD_INFRA_CONTAINER_ID=ce89bae7ebf0b4a62c08aad3db9eb1a913d490b4eea208dd5019ffe4522cdbc2;K8S_POD_UID=abb80a31-18a5-464a-a4e4-5b754950f121 Path:/opt/cni/bin StdinData:[123 34 97 114 103 115 34 58 123 34 115 101 114 118 105 99 101 77 101 115 104 77 111 100 101 34 58 34 108 105 110 107 101 114 100 34 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 107 117 98 101 114 110 101 116 101 115 34 58 123 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 90 90 90 45 109 101 114 98 114 105 100 103 101 45 99 110 105 45 107 117 98 101 99 111 110 102 105 103 34 125 44 34 110 97 109 101 34 58 34 107 56 115 45 112 111 100 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 101 114 98 114 105 100 103 101 45 99 110 105 34 125]}" func="cni-server.(*server).PodDeleted()" file="handlers.go:58"
    2022-12-10T17:22:03.426050Z	info	http: superfluous response.WriteHeader call from github.com/merbridge/merbridge/internal/cni-server.(*server).PodDeleted (handlers.go:68)
    time="2022-12-10T17:22:04Z" level=info msg="cni called delete with args: {ContainerID:4cbd7e03237e1c4152472178f734b43502ffde00e145e28e1d7fc190b6ee1728 Netns:/var/run/netns/cni-e449f8ef-c745-700e-e8f3-e9a0728c49b5 IfName:eth0 Args:K8S_POD_NAMESPACE=lens-metrics;K8S_POD_NAME=node-exporter-pvsxx;K8S_POD_INFRA_CONTAINER_ID=4cbd7e03237e1c4152472178f734b43502ffde00e145e28e1d7fc190b6ee1728;K8S_POD_UID=697d4eac-374f-4c62-9feb-914d2f036acc;IgnoreUnknown=1 Path:/opt/cni/bin StdinData:[123 34 97 114 103 115 34 58 123 34 115 101 114 118 105 99 101 77 101 115 104 77 111 100 101 34 58 34 108 105 110 107 101 114 100 34 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 107 117 98 101 114 110 101 116 101 115 34 58 123 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 90 90 90 45 109 101 114 98 114 105 100 103 101 45 99 110 105 45 107 117 98 101 99 111 110 102 105 103 34 125 44 34 110 97 109 101 34 58 34 107 56 115 45 112 111 100 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 101 114 98 114 105 100 103 101 45 99 110 105 34 125]}" func="cni-server.(*server).PodDeleted()" file="handlers.go:58"
    2022-12-10T17:22:04.408446Z	info	http: superfluous response.WriteHeader call from github.com/merbridge/merbridge/internal/cni-server.(*server).PodDeleted (handlers.go:68)
    time="2022-12-10T17:22:09Z" level=info msg="cni called delete with args: {ContainerID:0204449b6017e1efa1ab5e3acf91192a73b99f7cceb60bf53ab47b1fa0df4cfc Netns:/var/run/netns/cni-c316697d-789f-8019-0c2b-1ffec07156da IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=linkerd-viz;K8S_POD_NAME=tap-5f45486f68-mhdm7;K8S_POD_INFRA_CONTAINER_ID=0204449b6017e1efa1ab5e3acf91192a73b99f7cceb60bf53ab47b1fa0df4cfc;K8S_POD_UID=daa34ff7-4616-456b-a405-505fb4f09d02 Path:/opt/cni/bin StdinData:[123 34 97 114 103 115 34 58 123 34 115 101 114 118 105 99 101 77 101 115 104 77 111 100 101 34 58 34 108 105 110 107 101 114 100 34 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 107 117 98 101 114 110 101 116 101 115 34 58 123 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 90 90 90 45 109 101 114 98 114 105 100 103 101 45 99 110 105 45 107 117 98 101 99 111 110 102 105 103 34 125 44 34 110 97 109 101 34 58 34 107 56 115 45 112 111 100 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 101 114 98 114 105 100 103 101 45 99 110 105 34 125]}" func="cni-server.(*server).PodDeleted()" file="handlers.go:58"
    2022-12-10T17:22:09.431593Z	info	http: superfluous response.WriteHeader call from github.com/merbridge/merbridge/internal/cni-server.(*server).PodDeleted (handlers.go:68)
    

    2nd failing node details OS: linux (arm64) OS Image: Bottlerocket OS 1.11.1 (aws-k8s-1.24) Kernel version: 5.15.59 Container runtime: containerd://1.6.8+bottlerocket Kubelet version: v1.24.6-eks-4360b32 AWS EC2 instance type: m6gd.medium

    ** Merbridge logs

    2022-12-10T17:22:13.221412Z	warn	OS CA Cert could not be found for agent
    [ -f bpf/mb_connect.c ] && make -C bpf load || make -C bpf load-from-obj
    make[1]: Entering directory '/app/bpf'
    clang -O2 -g  -Wall -target bpf -I/usr/include/aarch64-linux-gnu  -DMESH=2 -DENABLE_IPV4=1 -DENABLE_IPV6=0 -c mb_connect.c -o mb_connect.o
    clang -O2 -g  -Wall -target bpf -I/usr/include/aarch64-linux-gnu  -DMESH=2 -DENABLE_IPV4=1 -DENABLE_IPV6=0 -c mb_get_sockopts.c -o mb_get_sockopts.o
    clang -O2 -g  -Wall -target bpf -I/usr/include/aarch64-linux-gnu  -DMESH=2 -DENABLE_IPV4=1 -DENABLE_IPV6=0 -c mb_redir.c -o mb_redir.o
    clang -O2 -g  -Wall -target bpf -I/usr/include/aarch64-linux-gnu  -DMESH=2 -DENABLE_IPV4=1 -DENABLE_IPV6=0 -c mb_sockops.c -o mb_sockops.o
    clang -O2 -g  -Wall -target bpf -I/usr/include/aarch64-linux-gnu  -DMESH=2 -DENABLE_IPV4=1 -DENABLE_IPV6=0 -c mb_bind.c -o mb_bind.o
    clang -O2 -g  -Wall -target bpf -I/usr/include/aarch64-linux-gnu  -DMESH=2 -DENABLE_IPV4=1 -DENABLE_IPV6=0 -c mb_sendmsg.c -o mb_sendmsg.o
    clang -O2 -g  -Wall -target bpf -I/usr/include/aarch64-linux-gnu  -DMESH=2 -DENABLE_IPV4=1 -DENABLE_IPV6=0 -c mb_recvmsg.c -o mb_recvmsg.o
    clang -O2 -g  -Wall -target bpf -I/usr/include/aarch64-linux-gnu  -DMESH=2 -DENABLE_IPV4=1 -DENABLE_IPV6=0 -c mb_tc.c -o mb_tc.o
    sudo mount -t bpf bpf /sys/fs/bpf
    sudo mkdir -p /sys/fs/bpf/tc/globals
    [ -f /sys/fs/bpf/cookie_original_dst ] || sudo bpftool map create /sys/fs/bpf/cookie_original_dst type lru_hash key 8 value 24 entries 65535 name cookie_original_dst
    [ -f /sys/fs/bpf/local_pod_ips ] || sudo bpftool map create /sys/fs/bpf/local_pod_ips type hash key 16 value 244 entries 1024 name local_pod_ips
    [ -f /sys/fs/bpf/process_ip ] || sudo bpftool map create /sys/fs/bpf/process_ip type lru_hash key 4 value 4 entries 1024 name process_ip
    [ -f /sys/fs/bpf/cgroup_info_map ] || sudo bpftool map create /sys/fs/bpf/cgroup_info_map type lru_hash key 8 value 32 entries 1024 name cgroup_info_map
    [ -f /sys/fs/bpf/mark_pod_ips_map ] || sudo bpftool map create /sys/fs/bpf/mark_pod_ips_map type hash key 4 value 16 entries 65535 name mark_pod_ips_map
    sudo bpftool -m prog loadall mb_connect.o /sys/fs/bpf/connect \
    	map name cookie_original_dst pinned /sys/fs/bpf/cookie_original_dst \
    	map name local_pod_ips pinned /sys/fs/bpf/local_pod_ips \
    	map name mark_pod_ips_map pinned /sys/fs/bpf/mark_pod_ips_map \
    	map name process_ip pinned /sys/fs/bpf/process_ip \
    	map name cgroup_info_map pinned /sys/fs/bpf/cgroup_info_map
    [ -f /sys/fs/bpf/pair_original_dst ] || sudo bpftool map create /sys/fs/bpf/pair_original_dst type lru_hash key 36 value 24 entries 65535 name pair_original_dst
    [ -f /sys/fs/bpf/sock_pair_map ] || sudo bpftool map create /sys/fs/bpf/sock_pair_map type sockhash key 36 value 4 entries 65535 name sock_pair_map
    sudo bpftool -m prog load mb_sockops.o /sys/fs/bpf/sockops \
    	map name cookie_original_dst pinned /sys/fs/bpf/cookie_original_dst \
    	map name process_ip pinned /sys/fs/bpf/process_ip \
    	map name pair_original_dst pinned /sys/fs/bpf/pair_original_dst \
    	map name sock_pair_map pinned /sys/fs/bpf/sock_pair_map
    sudo bpftool -m prog load mb_get_sockopts.o /sys/fs/bpf/get_sockopts \
    	map name pair_original_dst pinned /sys/fs/bpf/pair_original_dst
    sudo bpftool -m prog load mb_redir.o /sys/fs/bpf/redir \
    	map name sock_pair_map pinned /sys/fs/bpf/sock_pair_map
    sudo bpftool -m prog load mb_bind.o /sys/fs/bpf/bind
    sudo bpftool -m prog loadall mb_sendmsg.o /sys/fs/bpf/sendmsg \
    	map name cookie_original_dst pinned /sys/fs/bpf/cookie_original_dst \
    	map name mark_pod_ips_map pinned /sys/fs/bpf/mark_pod_ips_map \
    	map name cgroup_info_map pinned /sys/fs/bpf/cgroup_info_map
    sudo bpftool -m prog loadall mb_recvmsg.o /sys/fs/bpf/recvmsg \
    	map name cookie_original_dst pinned /sys/fs/bpf/cookie_original_dst \
    	map name mark_pod_ips_map pinned /sys/fs/bpf/mark_pod_ips_map \
    	map name cgroup_info_map pinned /sys/fs/bpf/cgroup_info_map
    make[1]: Leaving directory '/app/bpf'
    time="2022-12-10T17:22:15Z" level=info msg="Copied /app/merbridge-cni to /host/opt/cni/bin." func="cni-server.copyBinaries()" file="install.go:363"
    time="2022-12-10T17:22:15Z" level=info msg="write kubeconfig file /host/etc/cni/net.d/ZZZ-merbridge-cni-kubeconfig with: \n# Kubeconfig file for Merbridge CNI plugin.\napiVersion: v1\nkind: Config\nclusters:\n- name: local\n  cluster:\n    server: https://[10.100.0.1]:443\n    insecure-skip-tls-verify: true\nusers:\n- name: merbridge-cni\n  user:\n    token: \"<redacted>\"\ncontexts:\n- name: merbridge-cni-context\n  context:\n    cluster: local\n    user: merbridge-cni\ncurrent-context: merbridge-cni-context\n" func="cni-server.createKubeconfigFile()" file="install.go:453"
    time="2022-12-10T17:22:15Z" level=info msg="CNI config file /host/etc/cni/net.d/01-linkerd-cni.conf exists. Proceeding." func="cni-server.getCNIConfigFilepath()" file="install.go:310"
    time="2022-12-10T17:22:15Z" level=info msg="Renaming /host/etc/cni/net.d/01-linkerd-cni.conf extension to .conflist" func="cni-server.writeCNIConfig()" file="install.go:259"
    time="2022-12-10T17:22:15Z" level=info msg="Created CNI config /host/etc/cni/net.d/01-linkerd-cni.conflist" func="cni-server.writeCNIConfig()" file="install.go:267"
    time="2022-12-10T17:22:15Z" level=info msg="Pod Watcher Ready" func="controller.RunLocalPodController()" file="pod.go:53"
    make -C bpf attach
    make[1]: Entering directory '/app/bpf'
    sudo bpftool cgroup attach /sys/fs/cgroup/unified connect4 pinned /sys/fs/bpf/connect/cgroup_connect4
    sudo bpftool cgroup attach /sys/fs/cgroup/unified sock_ops pinned /sys/fs/bpf/sockops
    sudo bpftool cgroup attach /sys/fs/cgroup/unified getsockopt pinned /sys/fs/bpf/get_sockopts
    sudo bpftool prog attach pinned /sys/fs/bpf/redir msg_verdict pinned /sys/fs/bpf/sock_pair_map
    sudo bpftool cgroup attach /sys/fs/cgroup/unified bind4 pinned /sys/fs/bpf/bind
    sudo bpftool cgroup attach /sys/fs/cgroup/unified sendmsg4 pinned /sys/fs/bpf/sendmsg/cgroup_sendmsg4
    sudo bpftool cgroup attach /sys/fs/cgroup/unified recvmsg4 pinned /sys/fs/bpf/recvmsg/cgroup_recvmsg4
    make[1]: Leaving directory '/app/bpf'
    time="2022-12-10T17:22:16Z" level=info msg="cni called delete with args: {ContainerID:47ef94afa54e33404bf90eef472a8c08dd5c9c99edca255ce75802d422d34122 Netns:/var/run/netns/cni-acf182b9-6e9d-9d40-37ad-dfab01b134f8 IfName:eth0 Args:K8S_POD_INFRA_CONTAINER_ID=47ef94afa54e33404bf90eef472a8c08dd5c9c99edca255ce75802d422d34122;K8S_POD_UID=45ae086f-34bf-4fad-82e6-c6f21d93b302;IgnoreUnknown=1;K8S_POD_NAMESPACE=lens-metrics;K8S_POD_NAME=node-exporter-b2qn7 Path:/opt/cni/bin StdinData:[123 34 97 114 103 115 34 58 123 34 115 101 114 118 105 99 101 77 101 115 104 77 111 100 101 34 58 34 108 105 110 107 101 114 100 34 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 107 117 98 101 114 110 101 116 101 115 34 58 123 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 90 90 90 45 109 101 114 98 114 105 100 103 101 45 99 110 105 45 107 117 98 101 99 111 110 102 105 103 34 125 44 34 110 97 109 101 34 58 34 107 56 115 45 112 111 100 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 101 114 98 114 105 100 103 101 45 99 110 105 34 125]}" func="cni-server.(*server).PodDeleted()" file="handlers.go:58"
    2022-12-10T17:22:16.196130Z	info	http: superfluous response.WriteHeader call from github.com/merbridge/merbridge/internal/cni-server.(*server).PodDeleted (handlers.go:68)
    time="2022-12-10T17:22:16Z" level=info msg="cni called delete with args: {ContainerID:312ded126900f5a479b72d5a27e5940cf8f47ddd249def0c508b740480c68d2d Netns:/var/run/netns/cni-124c5911-ae72-6fc8-e21b-0a94cb7aa830 IfName:eth0 Args:K8S_POD_INFRA_CONTAINER_ID=312ded126900f5a479b72d5a27e5940cf8f47ddd249def0c508b740480c68d2d;K8S_POD_UID=267ff840-b883-40e9-bdfc-0bfb8d732f9c;IgnoreUnknown=1;K8S_POD_NAMESPACE=linkerd-viz;K8S_POD_NAME=tap-5f45486f68-dvcww Path:/opt/cni/bin StdinData:[123 34 97 114 103 115 34 58 123 34 115 101 114 118 105 99 101 77 101 115 104 77 111 100 101 34 58 34 108 105 110 107 101 114 100 34 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 107 117 98 101 114 110 101 116 101 115 34 58 123 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 90 90 90 45 109 101 114 98 114 105 100 103 101 45 99 110 105 45 107 117 98 101 99 111 110 102 105 103 34 125 44 34 110 97 109 101 34 58 34 107 56 115 45 112 111 100 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 101 114 98 114 105 100 103 101 45 99 110 105 34 125]}" func="cni-server.(*server).PodDeleted()" file="handlers.go:58"
    2022-12-10T17:22:16.278767Z	info	http: superfluous response.WriteHeader call from github.com/merbridge/merbridge/internal/cni-server.(*server).PodDeleted (handlers.go:68)
    time="2022-12-10T17:22:18Z" level=info msg="cni called delete with args: {ContainerID:0c101962bb6b3609cd5e2c6b3a2738593e02cac04bb690a9a4a20d80f332b532 Netns:/var/run/netns/cni-84b74e60-7ba3-8897-9460-4fcea9976adb IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=ebs-csi-node-rc66g;K8S_POD_INFRA_CONTAINER_ID=0c101962bb6b3609cd5e2c6b3a2738593e02cac04bb690a9a4a20d80f332b532;K8S_POD_UID=89ac567a-cd56-4fda-8db4-e0c2bb71343d Path:/opt/cni/bin StdinData:[123 34 97 114 103 115 34 58 123 34 115 101 114 118 105 99 101 77 101 115 104 77 111 100 101 34 58 34 108 105 110 107 101 114 100 34 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 107 117 98 101 114 110 101 116 101 115 34 58 123 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 90 90 90 45 109 101 114 98 114 105 100 103 101 45 99 110 105 45 107 117 98 101 99 111 110 102 105 103 34 125 44 34 110 97 109 101 34 58 34 107 56 115 45 112 111 100 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 101 114 98 114 105 100 103 101 45 99 110 105 34 125]}" func="cni-server.(*server).PodDeleted()" file="handlers.go:58"
    2022-12-10T17:22:18.179878Z	info	http: superfluous response.WriteHeader call from github.com/merbridge/merbridge/internal/cni-server.(*server).PodDeleted (handlers.go:68)
    time="2022-12-10T17:22:18Z" level=info msg="cni called delete with args: {ContainerID:b045e0a7ac6c237065baf41c2179fb5e6bc4579297d31d388c8dc58b1a8864d2 Netns:/var/run/netns/cni-5e7e2cb4-1377-915e-d391-c65c9a454888 IfName:eth0 Args:K8S_POD_NAMESPACE=linkerd-viz;K8S_POD_NAME=prometheus-74dd4ffb74-c7chh;K8S_POD_INFRA_CONTAINER_ID=b045e0a7ac6c237065baf41c2179fb5e6bc4579297d31d388c8dc58b1a8864d2;K8S_POD_UID=509cb2fb-0ffb-495b-8224-9f2140ff8a04;IgnoreUnknown=1 Path:/opt/cni/bin StdinData:[123 34 97 114 103 115 34 58 123 34 115 101 114 118 105 99 101 77 101 115 104 77 111 100 101 34 58 34 108 105 110 107 101 114 100 34 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 107 117 98 101 114 110 101 116 101 115 34 58 123 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 90 90 90 45 109 101 114 98 114 105 100 103 101 45 99 110 105 45 107 117 98 101 99 111 110 102 105 103 34 125 44 34 110 97 109 101 34 58 34 107 56 115 45 112 111 100 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 101 114 98 114 105 100 103 101 45 99 110 105 34 125]}" func="cni-server.(*server).PodDeleted()" file="handlers.go:58"
    2022-12-10T17:22:18.285592Z	info	http: superfluous response.WriteHeader call from github.com/merbridge/merbridge/internal/cni-server.(*server).PodDeleted (handlers.go:68)
    time="2022-12-10T17:22:18Z" level=info msg="Restarting Merbridge CNI installer..." func="cni-server.(*Installer).Run()" file="install.go:113"
    time="2022-12-10T17:22:19Z" level=info msg="Copied /app/merbridge-cni to /host/opt/cni/bin." func="cni-server.copyBinaries()" file="install.go:363"
    time="2022-12-10T17:22:19Z" level=info msg="write kubeconfig file /host/etc/cni/net.d/ZZZ-merbridge-cni-kubeconfig with: \n# Kubeconfig file for Merbridge CNI plugin.\napiVersion: v1\nkind: Config\nclusters:\n- name: local\n  cluster:\n    server: https://[10.100.0.1]:443\n    insecure-skip-tls-verify: true\nusers:\n- name: merbridge-cni\n  user:\n    token: \"<redacted>\"\ncontexts:\n- name: merbridge-cni-context\n  context:\n    cluster: local\n    user: merbridge-cni\ncurrent-context: merbridge-cni-context\n" func="cni-server.createKubeconfigFile()" file="install.go:453"
    time="2022-12-10T17:22:19Z" level=info msg="CNI config file /host/etc/cni/net.d/01-linkerd-cni.conflist exists. Proceeding." func="cni-server.getCNIConfigFilepath()" file="install.go:310"
    time="2022-12-10T17:22:19Z" level=info msg="Created CNI config /host/etc/cni/net.d/01-linkerd-cni.conflist" func="cni-server.writeCNIConfig()" file="install.go:267"
    time="2022-12-10T17:22:29Z" level=info msg="cni called delete with args: {ContainerID:7c211e35501935b10edbc48477aef2fc3cba653c4d112d08032a9b34a607decf Netns:/var/run/netns/cni-68cf0c92-feb9-625c-5d8b-e6dd047542c2 IfName:eth0 Args:IgnoreUnknown=1;K8S_POD_NAMESPACE=linkerd-viz;K8S_POD_NAME=tap-5f45486f68-dvcww;K8S_POD_INFRA_CONTAINER_ID=7c211e35501935b10edbc48477aef2fc3cba653c4d112d08032a9b34a607decf;K8S_POD_UID=267ff840-b883-40e9-bdfc-0bfb8d732f9c Path:/opt/cni/bin StdinData:[123 34 97 114 103 115 34 58 123 34 115 101 114 118 105 99 101 77 101 115 104 77 111 100 101 34 58 34 108 105 110 107 101 114 100 34 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 107 117 98 101 114 110 101 116 101 115 34 58 123 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 90 90 90 45 109 101 114 98 114 105 100 103 101 45 99 110 105 45 107 117 98 101 99 111 110 102 105 103 34 125 44 34 110 97 109 101 34 58 34 107 56 115 45 112 111 100 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 101 114 98 114 105 100 103 101 45 99 110 105 34 125]}" func="cni-server.(*server).PodDeleted()" file="handlers.go:58"
    2022-12-10T17:22:29.548978Z	info	http: superfluous response.WriteHeader call from github.com/merbridge/merbridge/internal/cni-server.(*server).PodDeleted (handlers.go:68)
    time="2022-12-10T17:22:30Z" level=info msg="cni called delete with args: {ContainerID:b082ca4a684f5381ea0a7f510b7a5ef3e401bd26cec3185d37e62f04e3ca674f Netns:/var/run/netns/cni-e32ec8c1-ec2d-491d-fa87-387ed183593c IfName:eth0 Args:K8S_POD_UID=89ac567a-cd56-4fda-8db4-e0c2bb71343d;IgnoreUnknown=1;K8S_POD_NAMESPACE=kube-system;K8S_POD_NAME=ebs-csi-node-rc66g;K8S_POD_INFRA_CONTAINER_ID=b082ca4a684f5381ea0a7f510b7a5ef3e401bd26cec3185d37e62f04e3ca674f Path:/opt/cni/bin StdinData:[123 34 97 114 103 115 34 58 123 34 115 101 114 118 105 99 101 77 101 115 104 77 111 100 101 34 58 34 108 105 110 107 101 114 100 34 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 107 117 98 101 114 110 101 116 101 115 34 58 123 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 90 90 90 45 109 101 114 98 114 105 100 103 101 45 99 110 105 45 107 117 98 101 99 111 110 102 105 103 34 125 44 34 110 97 109 101 34 58 34 107 56 115 45 112 111 100 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 101 114 98 114 105 100 103 101 45 99 110 105 34 125]}" func="cni-server.(*server).PodDeleted()" file="handlers.go:58"
    2022-12-10T17:22:30.203695Z	info	http: superfluous response.WriteHeader call from github.com/merbridge/merbridge/internal/cni-server.(*server).PodDeleted (handlers.go:68)
    time="2022-12-10T17:22:30Z" level=info msg="cni called delete with args: {ContainerID:54527144c307e5f3d3038396d90519b21adb42c5b2662c38dd94098a4cba0b45 Netns:/var/run/netns/cni-7787beec-0b39-027c-b787-3b6a6ea624d9 IfName:eth0 Args:K8S_POD_NAME=prometheus-74dd4ffb74-c7chh;K8S_POD_INFRA_CONTAINER_ID=54527144c307e5f3d3038396d90519b21adb42c5b2662c38dd94098a4cba0b45;K8S_POD_UID=509cb2fb-0ffb-495b-8224-9f2140ff8a04;IgnoreUnknown=1;K8S_POD_NAMESPACE=linkerd-viz Path:/opt/cni/bin StdinData:[123 34 97 114 103 115 34 58 123 34 115 101 114 118 105 99 101 77 101 115 104 77 111 100 101 34 58 34 108 105 110 107 101 114 100 34 125 44 34 99 110 105 86 101 114 115 105 111 110 34 58 34 48 46 51 46 49 34 44 34 107 117 98 101 114 110 101 116 101 115 34 58 123 34 107 117 98 101 99 111 110 102 105 103 34 58 34 47 101 116 99 47 99 110 105 47 110 101 116 46 100 47 90 90 90 45 109 101 114 98 114 105 100 103 101 45 99 110 105 45 107 117 98 101 99 111 110 102 105 103 34 125 44 34 110 97 109 101 34 58 34 107 56 115 45 112 111 100 45 110 101 116 119 111 114 107 34 44 34 116 121 112 101 34 58 34 109 101 114 98 114 105 100 103 101 45 99 110 105 34 125]}" func="cni-server.(*server).PodDeleted()" file="handlers.go:58"
    2022-12-10T17:22:30.285416Z	info	http: superfluous response.WriteHeader call from github.com/merbridge/merbridge/internal/cni-server.(*server).PodDeleted (handlers.go:68)
    

    image

    Version

    OS: `linux (arm64)`
    OS Image: `Bottlerocket OS 1.11.1 (aws-k8s-1.24)`
    Kernel version: `5.15.59`
    

    Probably related to #218

EaseMesh is a service mesh that is compatible with the Spring Cloud ecosystem.
EaseMesh is a service mesh that is compatible with the Spring Cloud ecosystem.

A service mesh implementation for connecting, control, and observe services in spring-cloud.

Jan 4, 2023
The Consul API Gateway is a dedicated ingress solution for intelligently routing traffic to applications running on a Consul Service Mesh.

The Consul API Gateway is a dedicated ingress solution for intelligently routing traffic to applications running on a Consul Service Mesh.

Dec 14, 2022
Meshery, the service mesh management plane
Meshery, the service mesh management plane

Meshery is the multi-service mesh management plane offering lifecycle, configuration, and performance management of service meshes and their workloads.

Jan 5, 2023
Rpcx-framework - An RPC microservices framework based on rpcx, simple and easy to use, ultra fast and efficient, powerful, service discovery, service governance, service layering, version control, routing label registration.

RPCX Framework An RPC microservices framework based on rpcx. Features: simple and easy to use, ultra fast and efficient, powerful, service discovery,

Jan 5, 2022
Wrapper to easily generate "X-Request-Auth" header for Mesh sites in golang.

hawk mesh go ?? ?? Description Wrapper to easily generate "X-Request-Auth" header for Mesh sites in golang. Based on hawk-go. Getting Started Import t

Dec 4, 2022
📕 twtxt is a Self-Hosted, Twitter™-like Decentralised microBlogging platform. No ads, no tracking, your content, your data!
📕 twtxt is a Self-Hosted, Twitter™-like Decentralised microBlogging platform. No ads, no tracking, your content, your data!

twtxt ?? twtxt is a Self-Hosted, Twitter™-like Decentralised micro-Blogging platform. No ads, no tracking, your content, your data! Technically twtxt

Jul 26, 2021
Collection of personal Dapr demos (bindings, state, pub/sub, service-to-service invocation)

Dapr demos Collection of personal Dapr demos. Note, some of these demos require latest version of Dapr, Ingress gateway, Observability components, or

Dec 10, 2022
A service showing your IP.

showip A service showing your IP. Simple network discovery service to help applications behind NAT routers discover their own external IP address. Usa

Feb 13, 2022
Best microservices framework in Go, like alibaba Dubbo, but with more features, Scale easily.
Best microservices framework in Go, like alibaba Dubbo, but with more features, Scale easily.

Best microservices framework in Go, like alibaba Dubbo, but with more features, Scale easily.

Dec 30, 2022
An example microservice demo using kubernetes concepts like deployment, services, persistent volume and claims, secrets and helm chart

Docker vs Kubernetes Docker Kubernetes container tech, isolated env for apps infra management, multiple containers automated builds and deploy apps -

Dec 13, 2021
Cloud-native and easy-to-use application management platform | 云原生且易用的应用管理平台
Cloud-native and easy-to-use application management platform | 云原生且易用的应用管理平台

Website • Documentation What is NEW! August 24, 2020 ,Rainbond 5.2 Stable version is officially released View Release Rainbond Introduction Cloud nati

Dec 29, 2022
This is a shopping basket workshop that shows how to use KrakenD API Gateway.
This is a shopping basket workshop that shows how to use KrakenD API Gateway.

Go Restful Microservices and KrakenD API Gateway Workshop This is a shopping basket workshop that shows how to use KrakenD API Gateway. Consist of 5 m

Jan 3, 2023
Demo for my talk at ArgoCon '21 showing how to use Go to create and submit dynamic Argo Workflows.
Demo for my talk at ArgoCon '21 showing how to use Go to create and submit dynamic Argo Workflows.

argocon-21-demo Demo for my talk at ArgoCon '21 showing how to use Go to create and submit dynamic Argo Workflows. This repo implements a Go-based CLI

Oct 12, 2022
Services-inoeg - The Kiebitz Backend Services. Still a work-in-progess, use with care!

Kiebitz Services This repository contains Kiebitz's backend services: A storage service that stores encrypted user & operator settings and temporary d

Jan 19, 2022
Just a quick demo of how you can use automatically generated protobuffer and gRPC code from buf.build

buf.build demo The purpose of this repository is to demonstrate how to use the services offered by buf.build for hosting protobuffer definitions and a

Jan 4, 2022
Golang Microservice making use of protobuf and gRPC as the underlying transport protocol.

Go-Microservices Golang Microservice making use of protobuf and gRPC as the underlying transport protocol. I will be building a generic microservice,

Jan 5, 2022
Assignment2 - A shared project making use of microservice architecture

This project is a shared project making use of microservice architecture, API's and a simple frontend to implement a start-up new concept called EduFi. The concept combines education and financial systems to create profit from studying.

Jan 26, 2022
Micro-service framework in Go
Micro-service framework in Go

Kite Micro-Service Framework Kite is a framework for developing micro-services in Go. Kite is both the name of the framework and the micro-service tha

Jan 9, 2023
NewSQL distributed storage database based on micro service framework
NewSQL distributed storage database based on micro service framework

QLite 是基于微服务的 NewSQL 型数据库系统,与传统的一体化数据库不同,该系统将本该内置的多种数据结构(STL)拆分成多个服务模块,每个模块都是独立的一个节点,每个节点都与其主网关进行连接,从而形成分布式存储结构。

Jun 19, 2022