Semantic Versioning (or SemVer) is a universally accepted format to tag your release packages/modules etc. It has vX.Y.Z format where X is a major version number, Y is a minor version number and Z is patch version number (translated as vMajor.Minor.Patch).

Also keeping consistent with Upstream fork versioning scheme, which will help avoid confusion. Doing this before anyone has taken dependency on releases.

Need to update Kubernetes version for security fixes (mentioned in issue #30). Upstream fork used v1.13.0-alpha, which has some security issues. We need at least v1.19.15 to fix all these:

Part of what makes this complicated is usage from deprecated Kubernetes packages from which no longer exists:

1. k8s.io/apiserver/pkg/util/logs moved in https://github.com/kubernetes/kubernetes/pull/48581 https://github.com/kubernetes/kubernetes/pull/79198 and even ended up in kubectl repo, only to be removed later: https://github.com/kubernetes/kubectl/commit/466b48353181e0479672d43c69fb6d42a0c9264f

2. k8s.io/kubernetes/pkg/util/dbus removed in: https://github.com/kubernetes/kubernetes/pull/81517

3. k8s.io/kubernetes/pkg/version/verflag removed in: https://github.com/kubernetes/kubernetes/pull/68922

Workarounds:

1. Switching to klog from glog, which is an up-to-date replacement. k8s.io/component-base/logs pairs with it and replaces k8s.io/apiserver/pkg/util/logs.
2. Following their example which relies on updated iptable package: https://github.com/kubernetes/kubernetes/pull/81517
3. k8s.io/component-base/version/verflag replaces this.

Also updating other dependancies while we are at it.

Thanks guys for letting me add this PR for your repo and I hope this helps @mattstam 🙏☕️❤️ also I think we can do separate pipeline for ghcr images as build and publish I don't quiet know the exact use-case so we can chat about it.

This PR focuses on adding end-to-end pipeline and seems like currently tests are not working so we can do test PR separately and then all we need to do is enable the last line of this build fencing file with the test fixture.

• 1 file with lint, build and test will make sure all check happens at once and like the screenshot shows below it will work as follows:

Thanks

The previous update to distroless base image was not compatible with all architectures.

In https://github.com/kubernetes/release/tree/master/images/build/distroless-iptables they publish for each architecture. However, staging-k8s.gcr.io is not publicly accessible. Ran the same make all-push but pointing at a AKS base image CR 'baseosscr.azurecr.io' which can be pulled from anywhere.

Roughly the same as @rikatz's rationale:

we need to reduce costs (and by reducing costs, reducing image size) we need to use something with a smaller footprint and reduce the amount of packages, and as a result the amount of CVEs in image package. kube-proxy exists in all nodes (unless you use a CNI that implements its features) so it is a good candidate

Same as kube-proxy and also keeping in sync with what they v1 did: https://github.com/kubernetes-sigs/ip-masq-agent/commit/cf9563a9605c65fc9d67dc8391d61bad89fc3295

klog has slightly different initialization behavior from the previously used glog. klog.InitFlags() must be called at startup time to handle the verbosity flags (e.g. --v=2).

With this change, one docker manifest for multiple platforms is used when the main image tag <registry>/<path>/<bin>:<version>:

This means that the appropriate image will be used depending on the platform that pulls it, allowing for full multi-arch usage.

To use simply run make all-manifest create a manifest for all platforms listed in the Makefile.

(also adding IBM's arch iptables base image)

Bunch of miscellaneous changes:

• Reference MCR image in examples
• Switch to -reconciled since it matches what cloud provider will likely use for maximum clarity.
• Add some missing content in README
• CHANGELOG references PR number, which you would have to know before hand. Keep formatting on releases page but removing that content from CHANGELOG so it's practical to update once per PR.

💡☕️ oh, a quick note: this versioning might not play nice with the release (publish mcr workflow) because of the change log action, so I would recommend if you want to keep “v” you could add that in your publishing tag file.

• quickest solution will be here add “v” as character before the versioning for “VERSION” variable: https://github.com/Azure/ip-masq-agent-v2/blob/master/.github/workflows/build-publish-mcr.yml#L66
• Change this line 66 from REGISTRY=${{ secrets.AZURE_REGISTRY_SERVER }}/public/aks VERSION=${{ steps.changelog_reader.outputs.version }} make push to REGISTRY=${{ secrets.AZURE_REGISTRY_SERVER }}/public/aks VERSION=v${{ steps.changelog_reader.outputs.version }} make push
• Changed bit is: VERSION=v${{ steps.changelog_reader.outputs.version }} make push added v so keep change-log as without v.

FYI and Cc: @peterbom in case I am not around.

Thank you @Tatsinnit for this! 🙏

Need to update Kubernetes version for security fixes. Part of what makes this difficult is usage from deprecated Kubernetes packages from which no longer exists.

k8s.io/apiserver/pkg/util/logs moved in https://github.com/kubernetes/kubernetes/pull/48581 https://github.com/kubernetes/kubernetes/pull/79198 and even ended up in kubectl repo, only to be removed later: https://github.com/kubernetes/kubectl/commit/466b48353181e0479672d43c69fb6d42a0c9264f

k8s.io/kubernetes/pkg/util/dbus moved in: https://github.com/kubernetes/kubernetes/pull/81517

k8s.io/kubernetes/pkg/version/verflag moved in: https://github.com/kubernetes/kubernetes/pull/68922 Not critical enough to warrant an immediate replacement. Removing for now.

Since updating dependancies, also switching to klog (https://github.com/kubernetes/klog) from glog, which is an up-to-date replacement.

Going from kubernetes version v0.21.1 to v1.13.0-alpha which may help with https://github.com/Azure/ip-masq-agent-v2/issues/25. Also attempting to fix broken CI lint: https://staticsint.teams.cdn.office.net/evergreen-assets/safelinks/1/atp-safelinks.html

Repro

• Create a default AKS cluster with kube-proxy in iptables mode
• Create a deployment of busybox pods with 50 replicas kubectl create deployment busybox --image busybox --replicas 50
• Run this gist to create 100 services each with 500 backend ports (50,000 backend port rules) https://gist.github.com/tyler-lloyd/5f6260b7ce6089fe944c35194cb841f7
• Observe ip-masq-agent getting OOMKill'd

It's unclear if ip-masq-agent is actually running out of memory or if it is getting killed by the host when trying to sync its iptables rules.

Error seen

F0701 15:14:36.026972       1 ip-masq-agent.go:156] the daemon encountered an error: error syncing masquerade rules: error creating chain "IP-MASQ-AGENT": exit status 4: Another app is currently holding the xtables lock; still 4s 100000us time ahead to have a chance to grab the lock...
Another app is currently holding the xtables lock; still 3s 100000us time ahead to have a chance to grab the lock...
Another app is currently holding the xtables lock; still 2s 100000us time ahead to have a chance to grab the lock...
Another app is currently holding the xtables lock; still 1s 100000us time ahead to have a chance to grab the lock...
Another app is currently holding the xtables lock; still 0s 100000us time ahead to have a chance to grab the lock...
Another app is currently holding the xtables lock. Stopped waiting after 5s.
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0x1)
	/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1140 +0x8a
k8s.io/klog/v2.(*loggingT).output(0xd40e40, 0x3, 0x0, 0xc0000a85b0, 0x0, {0xa918ee, 0x1}, 0xc0001bc140, 0x0)
	/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1088 +0x66f
k8s.io/klog/v2.(*loggingT).printf(0x60, 0x8ae02108, 0x0, {0x0, 0x0}, {0x9410c5, 0x23}, {0xc0001bc140, 0x1, 0x1})
	/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:753 +0x1c5
k8s.io/klog/v2.Fatalf(...)
	/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1642
main.main()
	/src/cmd/ip-masq-agent-v2/ip-masq-agent.go:156 +0x408

goroutine 6 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0x0)
	/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:1283 +0x6a
created by k8s.io/klog/v2.init.0
	/go/pkg/mod/k8s.io/klog/[email protected]/klog.go:420 +0xfb

Hello, I'm running an aks cluster (with kubernetes v1.23.5) and I'd like to limit the logs generated by ip-masq-agent-v2 to errors. I've seen that I could use the -v flag and set it to 1 in the daemonset for example, but somehow the default value 2 is always restored.

That's what I'm doing: kubectl edit daemonset azure-ip-masq-agent -n kube-system

replace - --v=2 by - --v=1 save and exit

pods are recreated and it looks good, but a few minutes later, logs are coming again. If I redo kubectl edit daemonset azure-ip-masq-agent -n kube-system, I can see the that -v is set to 2 again.

Do you have any idea of what I can do there? Thanks