1. Adding no auto-fence label to the global-sidecar. It will not auto create fence for global-sidecar when enable fence at namespace level.
2. Auto created fence name equals svc.spec.selector instead of svc.name now. For example, for svc/details with spec.selector[app]: details-new, it will create fence/details in the previous version and will create fence/details-new now.

Lazyload supports dynamic updating of service dependencies via metric, but in some scenarios, users want to add some static service dependencies when lazy loading is enabled, such as all services under a namespace, or services with certain labels. We will carefully consider the actual requirements and implement similar functionality through fence's spec field.

related to https://github.com/slime-io/lazyload/pull/36

Support for enabling lazyload for services manually or automatically

Support for enabling lazyload for services, either manually or automatically, via the autoFence parameter. Enabling lazyload here refers to the creation of the serviceFence resource, which generates the Sidecar CR.

Support for specifying whether lazyload is globally enabled in automatic mode via the defaultCreateFence parameter.

The configuration is as follows

---
apiVersion: config.netease.com/v1alpha1
kind: SlimeBoot
metadata:
  name: lazyload
  namespace: mesh-operator
spec:
  module:
    - name: lazyload
      kind: lazyload
      enable: true
      general:
        autoFence: true # true for automatic mode, false for manual mode, default for manual mode
        defaultCreateFence: true # Default behaviour in auto mode, true to create servicefence, false to not create, default not to create
  # ...

Auto mode

Auto mode is entered when the autoFence parameter is true. The range of services enabled for lazyload in auto mode is adjusted by three dimensions.

Service Level - label slime.io/serviceFenced

• false: not auto enable

• true: auto enable

• other values or empty: use namespace level configuration

Namespace Level - label slime.io/serviceFenced

• false: not auto enable for this namespace
• true: auto enable for this namespace
• other values or empty: use global level configuration

Global Level - defaultCreateFence param of lazyload module

• false: not auto enable for all
• true: auto enable for all

Priority: Service Level > Namespace Level > Global Level

Note: ServiceFence that are auto generated are labeled with app.kubernetes.io/created-by=fence-controller, which enables state association changes. ServiceFence that do not match this Label are considered to be manually configured and are not affected by the above Label.

Example

Namespace testns has 3 services, svc1, svc2, svc3

• When autoFence is true and defaultCreateFence is true, three ServiceFence for the above services is auto generated
• Label ns with slime.io/serviceFenced: "false", all ServiceFence disappear
• Label svc1 with slime.io/serviceFenced: "true" , create ServiceFence for svc1
• Delete the labels on Namespace and Service: created the three ServiceFence

Sample configuration

apiVersion: v1
kind: Namespace
metadata:
  labels:
    istio-injection: enabled
    slime.io/serviceFenced: "false"
  name: testns
---
apiVersion: v1
kind: Service
metadata:
  annotations: {}
  labels:
    app: svc1
    service: svc1
    slime.io/serviceFenced: "true"
  name: svc1
  namespace: testns

Manual mode

When the autoFence parameter is false, lazyload is enabled in manual mode, requiring the user to create the ServiceFence resource manually. This enablement is Service level.

Problem Solved

Global-sidecar would calculate the destination address to be forwarded based on the request header "Slime-Orig-Dest" when forwarding the request previously. When the request header "Slime-Orig-Dest" is empty, forwarding is rejected.

This PR extends the use of the global-sidecar scenario. When the request header "Slime-Orig-Dest" is empty, global-sidecar uses the request.host and the port on which the global-sidecar accepts the request as the destinatiuon address. Then request forwards directly, without generating an accesslog metric.

This allows business traffic to be directed to the global-sidecar for correct forwarding when there is a problem with some business sidecarProxy. This implies some sense of disaster recovery capability.

Notes

• general.wormholePort defines requests of which service ports should be forwarded, connected by ";"
• replicas of global-sidecar can be defined in component.globalSidecar.replicas
• resources of global-sidecar can be defined in component.globalSidecar.resources

Usage Example

---
apiVersion: config.netease.com/v1alpha1
kind: SlimeBoot
metadata:
  name: lazyload
  namespace: mesh-operator
spec:
  image:
    pullPolicy: Always
    repository: registry.cn-hangzhou.aliyuncs.com/slimeio/slime-lazyload
    tag: master-78f3d09_linux_amd64-dirty_e09cde4
  module:
    - name: lazyload-test
      kind: lazyload
      enable: true
      general:
        wormholePort: # replace to your application service ports, and extend the list in case of multi ports
          - "9080"
          - "9090"
      global:
        misc:
          globalSidecarMode: cluster
          metricSourceType: accesslog
  component:
    globalSidecar:
      enable: true
      sidecarInject:
        enable: true # should be true
        mode: pod
        labels:
          sidecar.istio.io/inject: "true"
      replicas: 2
      resources:
        requests:
          cpu: 200m
          memory: 200Mi
        limits:
          cpu: 400m
          memory: 400Mi
      image:
        repository: registry.cn-hangzhou.aliyuncs.com/slimeio/slime-global-sidecar
        tag: master-5d96f4d_linux_amd64
      probePort: 28888

New feature

This feature supports adding the name of the dependent target service and a set of its regular matches, converted by a field domainAlias, to the sidecarscope.

message Fence {
  // ...  
  // domain alias rule
  repeated DomainAlias domainAlias = 5;
}

message DomainAlias {
  string pattern = 1;
  repeated string templates = 2;
}

Supports both static and dynamic dependent target service.

Usage example

Add domainAlias with single inverted commas in fence module config

apiVersion: config.netease.com/v1alpha1
kind: SlimeBoot
metadata:
  name: lazyload
  namespace: mesh-operator
spec:
  module:
    - name: lazyload-test
      kind: lazyload
      enable: true
      general:
        wormholePort: # replace to your application service ports, and extend the list in case of multi ports
          - "9080"
        domainAliases: 
          - pattern: '(?P<service>[^\.]+)\.(?P<namespace>[^\.]+)\.svc\.cluster\.local$'
            templates:
              - "$namespace.$service.mailsaas"
  #...

serviceFence example

apiVersion: microservice.slime.io/v1alpha1
kind: ServiceFence
metadata:
  name: ratings
  namespace: default
spec:
  enable: true
  host:
    details.default.svc.cluster.local: # static dependent service
      stable: {}
status:
  domains:
    default.details.mailsaas: # static dependent service converted result
      hosts:
      - default.details.mailsaas
    default.productpage.mailsaas: # dynamic dependent service converted result
      hosts:
      - default.productpage.mailsaas
    details.default.svc.cluster.local:
      hosts:
      - details.default.svc.cluster.local
    productpage.default.svc.cluster.local:
      hosts:
      - productpage.default.svc.cluster.local
  metricStatus:
    '{destination_service="productpage.default.svc.cluster.local"}': "1" # dynamic dependent service

sidecar example

apiVersion: networking.istio.io/v1beta1
kind: Sidecar
metadata:
  name: ratings
  namespace: default
spec:
  egress:
  - hosts:
    - '*/default.details.mailsaas' # static dependent service converted result
    - '*/default.productpage.mailsaas' # dynamic dependent service converted result
    - '*/details.default.svc.cluster.local'
    - '*/productpage.default.svc.cluster.local'
    - istio-system/*
    - mesh-operator/*
  workloadSelector:
    labels:
      app: ratings

If service A requests a service out of cluster, like baidu.com, Lazyload will add '*/baidu.com.svc.cluster.local' to sidecar A in previous version, which is wrong. Now Lazyload can determine whether to add suffix svc.cluster.local or cluster.local according to service list in cluster. Then Lazyload will add '*/baidu.com' to sidecar A now.

In cluseter mode, all namespaces in the service mesh can use Lazyload, no need to explicitly specify a list of namespace like the namespace mode anymore.

In istio 1.12+, the way of metadata_exchange filters insertion is changed, related pr Add metadata exchange as native code. So global-sidecar will be inserted these filters.

Then prometheus metric, what we want is like service A -> service B, will be service A -> global-sidecar and global-sidecar -> service B. Lazyload cannot get right relationships anymore in previous version with prometheus metric mode.

In this pr, we remove metadata_exchange filters from global-sidecar via the envoyfilter named global-sidecar-metadata-exchange-remove, and this problem is solved.

Note

All versions of 1.12+ istio (latest version 1.13.2 so far) has problem with envoyfilter remove operation, causing lazyload with prometheus metric cannot run on these versions, related issue EnvoyFilter VIRTUAL_HOST REMOVE patch operation will destroy route configuration #36357. And the last bugfix commit is fix filter removal for http filter at 2022.03.16, not contained by any istio realease.

We provide a new istiod image registry.cn-hangzhou.aliyuncs.com/slimeio/pilot:1.13.2-bugfix based on istio 1.13.2 in order to run lazyload with prometheus metric.

Preceding work Move fence module config from framework to module

• Migrate namespace, dispatches fields to fence module.
• Preferred use general field to fill fence module config. If general is empty, try to use deprecated fence field in slimeboot operator.
• Need valid name or kind of module to use lazyload.

related to https://github.com/slime-io/slime/pull/138

Step 1: Make dynamic service dependency info be durable When lazyload starting, it will generate cache contains dynamic service dependency info from existed ServiceFences, which contains useful info in status.metricStatus. After this change, it will never cause dynamic relationships lost when lazyload pod restarting or deleting. This step has been completed in this PR.

Step 2: Make dynamic service dependency info be time-sensitive In step 1, lazyload will never delete dynamic service dependency info in any time. Suppose a dynamic dependency is not invoked again for a long time after a certain occurrence, at which point we should have a mechanism to delete the dynamic dependency. One scenario is that all service invocations record the time of the most recent invocation, that we globally set a time threshold that triggers deletion, and we also allow users to specify a dedicated time threshold for specific services. These will be done in subsequent development.

related to https://github.com/slime-io/slime/pull/137

In previous version, namespace and service level ServiceFence auto generating is auto enable. Now we can disable them through new fields in fence config, like below. Default value is false.

{
	"general": {
		"wormholePort": [
			"9090"
		],
		"disableAutoFence": true
	},
	"name": "lazyload"
}

lazyload can read from accelog and prometheus but the examples all mode need to deploy global sidecar? can slimeboot_no_global_sidecar.yaml be used?

Backgroud

Metric source for lazyload: Prometheus and AccessLog. In Prometheus mode, the data is stored in Prometheus and the LazyLoad Controller queries the results in real time without storing this data, so this mode does not require a persistence transformation. In AccessLog mode, the global-sidecar sends the AccessLog containing the service dependencies to the LazyLoad Controller after it has completed the underhanded forwarding. The LazyLoad Controller receives and processes the data and stores it in memory in the form of a Map. Once the Controller is restarted, all in-memory data is lost. When the ServiceFence is updated, the Status.MetricStatus of the servicefence is cleared and the dynamic service dependencies are lost.

Thinking

Step 1 Simple Persistence

The reason for the problem is that the cache map is initialised with a null value, but in fact, status.metricStatus of each ServiceFence is a good record of persistence dynamic service dependency and can be initialised by reversing servicefence.status.metricStatus -> cache map to complete the initialization. The follow-up is still one-way cache map -> servicefence.status.metricStatus

Step 2 Time-sensitive Persistence

In Step 1, the contents of the cache map are never deleted. Suppose a dynamic dependency is not invoked again for a long time after a certain occurrence, at which point we should have a mechanism to delete the dynamic dependency. One scenario would be to record when the accesslog metric is generated, set a global time threshold that triggers deletion, and delete the metric when the condition is met. we could even consider allowing users to specify a dedicated time threshold for a particular service. One problem with this is that when there is a metric in the cache map, subsequent calls are direct and do not go through the global-sidecar, so that the same accesslog metric is not generated again. Wouldn't that mean that lazyload sees it as inactive, regardless of whether the call is active or not? Resulting in all metric being periodically deleted.

Design

Step 1 Simple Persistence

AccessLogConvertorConfig adds InitCache When initializing the producer in lazyload, add the contents of the initialized initCache to get the ServiceFence information with the help of DynamicClient provided by bootstrap.Environment. You only need to convert these info to initCache later.

Step 2 Time-sensitive Persistence

TODO

Update (2022.04.25)

It has been verified that Istio 1.13.3 has fixed all relevant problems. We currently recommend versions 1.13.3+, 1.11.3 or 1.10.4.

经过验证，Istio 1.13.3已经修复了所有相关问题。目前我们推荐使用的版本为1.13.3+, 1.11.3或1.10.4。

Update (2022.03.22)

There are still some bugs with Envoyfilter REMOVE operation in Istio releases 1.13.1 ~ 1.13.2. The last bugfix commit is fix filter removal for http filter at 2022.03.16, not contained by any istio realease.

Lazyload with prometheus metric cannot run on these versions, while lazyload with accesslog metric has no problem.

We provide a new istiod image registry.cn-hangzhou.aliyuncs.com/slimeio/pilot:1.13.2-bugfix based on istio 1.13.2 in order to run lazyload with prometheus metric.

This problem will be solved in future istio 1.13.3+ so far as we know.

Update (2022.03.08)

It has been verified that this issue has been fixed in istio 1.13. The current recommended versions of istio are

• 1.13 All official versions
• 1.11.0 ~ 1.11.3
• 1.10.0 ~ 1.10.4
• 1.9.0 ~ 1.9.8

Versions not recommended for use are

• 1.12 All versions
• 1.11.4+
• 1.10.5 ~ 1.10.6
• 1.9.9

经过验证，目前istio 1.13已经修复此问题。 目前推荐使用的Istio版本为：

• 1.13 所有正式版本
• 1.11.0 ~ 1.11.3
• 1.10.0 ~ 1.10.4
• 1.9.0 ~ 1.9.8

不推荐使用的版本为：

• 1.12 所有版本
• 1.11.4+
• 1.10.5 ~ 1.10.6
• 1.9.9

After testing, there are currently compatibility issues with the latest Istio version for Lazyload. The cause of the problem, most likely, is that the new code in Istio causes an error in EnvoyFilter's REMOVE operation on VIRTUAL_HOST. This will affect the EnvoyFilter/to-global-sidecar of Lazyload module from taking effect.

Issue of the Istio community to follow up on this issue EnvoyFilter VIRTUAL_HOST REMOVE patch operation will destroy route configuration #36357

Code PR suspected of introducing problems do not build catch all virtual host every time #35449

Theoretically, any version of Istio released after October 2, 2021 could have the problem. The versions currently known to be affected are

• 1.12.0，1.12.1
• 1.11.4，1.11.5
• 1.10.5，1.10.6
• 1.9.9

We are currently recommending versions 1.11.3 or 1.10.4. We will continue to follow up on this issue and update this issue when there is progress.

经过测试，目前懒加载与最新的Istio版本存在兼容性问题。问题的原因，很有可能是Istio的新代码导致EnvoyFilter对VIRTUAL_HOST的REMOVE操作出现错误。这会影响懒加载模块兜底EnvoyFilter to-global-sidecar的生效。

Istio社区跟进此问题的Issue EnvoyFilter VIRTUAL_HOST REMOVE patch operation will destroy route configuration #36357

怀疑引入问题的代码PR do not build catch all virtual host every time #35449

理论上，2021年10月2号之后发布的Istio版本都可能存在问题。目前已知受影响的版本为：

• 1.12.0，1.12.1
• 1.11.4，1.11.5
• 1.10.5，1.10.6
• 1.9.9

目前我们推荐使用的版本为1.11.3或1.10.4。我们会持续跟进这个问题，并在有进展时，及时更新本issue。

Currently lazyload supports k8s service. In some scenarios, erviceentry is also useful. Therefore, we will seriously consider the solutions to support lazyload enabled for serviceentry.