希望提供一个通用的配置下发的grpc协议通道，用于layotto与控制面的交互（可以参考go-control-plane）， 可以随意定义用户可以，随意的定义下发配置的策略内容

用途： 1.redis、kafka客户端流量管理控制，可能需要容灾切换、灰度发布策略切换 2.redis连接参数调整、安全弱密码自动切换

Dapr现状

dapr现在在bindings里面支持了aliyun OSS的实现，具体实现可以参照oss。 ​

dapr的Binding的分为Input和Out前者就是个类似于sub的能力，收到事件，调用回调函数。后者就是对后端的组件进行一些CRUD操作,如下： ​

const (
	GetOperation    OperationKind = "get"
	CreateOperation OperationKind = "create"
	DeleteOperation OperationKind = "delete"
	ListOperation   OperationKind = "list"
)

​

  // Invokes binding data to specific output bindings
  rpc InvokeBinding(InvokeBindingRequest) returns (InvokeBindingResponse) {}


// InvokeBindingRequest is the message to send data to output bindings
message InvokeBindingRequest {
  // The name of the output binding to invoke.
  string name = 1;

  // The data which will be sent to output binding.
  bytes data = 2;  //对象数据

  // The metadata passing to output binding components
  // 
  // Common metadata property:
  // - ttlInSeconds : the time to live in seconds for the message. 
  // If set in the binding definition will cause all messages to 
  // have a default time to live. The message ttl overrides any value
  // in the binding definition.
  map<string,string> metadata = 3;  //OSS的key存在这里面

  // The name of the operation type for the binding to invoke
  string operation = 4;  //增删改查
}

// InvokeBindingResponse is the message returned from an output binding invocation
message InvokeBindingResponse {
  // The data which will be sent to output binding.
  bytes data = 1;

  // The metadata returned from an external system
  map<string,string> metadata = 2;
}

InvokeBinding是一个Unary的rpc调用，这对于OSS这种支持大文件需要流式传输的功能肯定是没法支持的。 ​

方案

方案一：

​

增加InvokeService为stream的rpc调用： ​

  // InvokeService do rpc calls
  rpc InvokeService(InvokeServiceRequest) returns (InvokeResponse) {}

	改为：

	// InvokeService do rpc calls
  rpc InvokeService(stream InvokeServiceRequest) returns (stream InvokeResponse) {}

​

先给Dapr提了个Issue来跟踪： https://github.com/dapr/dapr/issues/3338

方案二：

本身dapr的binding的概念就不是一个很好的抽象，感觉就是什么都可以做。但又不知道应该用来做哪些东西。对于OSS这种流式传输的，我们也可以在pb中单独抽象出一类接口：

  // Get file with stream.
  rpc GetFile(GetFileRequest) returns (stream GetFileResponse) {}

  // Put file with stream.
  rpc PutFile(stream PutFileRequest) returns (google.protobuf.Empty) {}

message GetFileRequest {
  //
  string store_name = 1;
  // The name of the file or object want to get.
  string name = 2;
  // The metadata for user set.
  map<string,string> metadata = 3;
}

message GetFileResp {
  bytes data = 1;
}

message PutFileRequest {
  string store_name = 1;
  // The name of the file or object want to put.
  string name = 2;
  // The data which will be store.
  bytes data = 3;
  // The metadata for user set.
  map<string,string> metadata = 4;
}

​

@kevinten10 目前遇到了多语言维护成本高的痛点，在调研 sidecar 模式

但是遇到了问题:

1. 已经在 AWS 和 阿里云上用了原生 sidecar （AWS app mesh 和 阿里云 ASM，其实都是 envoy)，在开启了透明流量劫持的情况下, 再部署个 layotto sidecar 会不会冲突?
2. 已经有了原生 sidecar (envoy), 再部个 mosn 比较冗余。能否部署一个 layotto 独立sidecar，但是里面不带mosn？

需求细节：

• rpc 需要走 layotto sidecar，因为有一些 tracing、可观测性相关逻辑要自己写；
• 云上的 mesh 不好做扩展、不考虑把扩展逻辑做在 envoy 里（AWS envoy 想扩展得自己打镜像、再交给 AWS 托管，有额外的运维成本，不想这么做）
• 考虑私有云采用 sdk（因为要复用已有中间件，不可能换个语言重写一遍）云上用 sidecar 模式。值得学习

action:

• [ ] 把 layotto 拆个独立包，以不带 mosn 的方式启动 @seeflood
• [ ] 找个 AWS 环境，二进制运行，看看流量冲突问题 @kevinten10

What this PR does:

Which issue(s) this PR fixes:

Fixes #513

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

将COMPONENT NAME与具体的中间件类型解藕，在COMPONENT NAME块中KEY:VAKLUE对新增type:name (name为中间件类型)

What this PR does:

Using double buffer to solve the problem of sequencer local cache.

Which issue(s) this PR fixes:

Fixes #158

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

What this PR does:

add zookeeper lock

Which issue(s) this PR fixes:

Fixes #104

Special notes for your reviewer:

@seeflood

Does this PR introduce a user-facing change?:

What this PR does:

add redis cluster lock(red lock implement)

Which issue(s) this PR fixes:

Fixes #249

环境

preview server k8s yaml: https://github.com/LXPWing/preview-server-k8s-yml

线上实验室配置文件： https://github.com/LXPWing/mosn-tutorial/tree/master/layotto/layotto-with-bookinfo

bookinfo k8s yaml: https://github.com/LXPWing/layotto/tree/main/deploy/k8s/bookinfo

线上实验室演示：https://killercoda.com/lixingpeng/course/layotto/layotto-with-bookinfo

问题

preview server与bookinfo都部署在default namespace下， preview server yml中设置replicas: 1但实际部署中会超出1个pod且报错，同时preview server pod中的镜像数量会比yml中设置的多1个。bookinfo 的一些pod也会出现超出设置数量且报错。

尝试解决

preview server和bookinfo的namespace分离，但是preview serve一直处于ContainerCreating

Hi, This is IM notify API proposal.

What would you like to be added:

saas api: IM notify api

Why is this needed:

https://github.com/mosn/layotto/issues/712

Support situation:

|IM|Docs| |---|---| |Dingtalk| https://open.dingtalk.com/document/group/send-a-dingtalk-robot-message | |Dingtalk dapr| https://docs.dapr.io/zh-hans/reference/components-reference/supported-bindings/alicloud-dingtalk/ |

API spec:

// Saas API
// Send the message to IM.
rpc SendIM(SendIMRequest) returns (SendIMResponse) {}

------

// SendIMRequest is the message send to im.
message SendIMRequest {

  // The invoke Id.
  string id = 1;

  // The saas service name, like 'dingtalk'/'wechat'/'...'
  string saas_name = 2;

  // The api access token.
  string secret = 3;

  // The IM sender tag.
  string sender_id = 4;

  // The IM receivers tag.
  repeated string receivers = 5;

  // The IM message type, like 'text'/'table'/'...'
  string message_type = 6;

  // The IM message data.
  bytes message_data = 7;

  // The metadata which will be sent to IM components.
  map<string, string> metadata = 8;
}

message SendIMResponse {
  // The saas requestId.
  string request_id = 1;

  // The data response from IM service.
  bytes data = 2;

  // The metadata returned from IM service.
  map<string, string> metadata = 3;
}

字段设计考虑:

1. IM原生API非常复杂，难取交集

无论是dingtalk，还是wechat等IM工具。在原生API设计上都有很多自己独特的请求参数，这取决于IM自身的一些设计。

例如：

• 消息模板ID：有的IM具有模板，有的IM不需要模板即可发送。
• 收件人格式：有的IM发送给自生成的ID，有的则使用 用户名/手机号 等进行发送。

2. IM具有绑定性，弱移植性

一个公司，哪怕部署到混合云，大概率也只会使用一种IM工具，很少听说不同云的消息要发送给不同IM。

所以本身IM代码就不需要很强的可移植性，那么对于一些特有的字段，可以接受放在metadata中。

3. Runtime API取核心字段，其他字段放metadata

尽可能取多种IM都有的一些概念，将这些字段放在API中，其他字段放metadata。

对于一次IM发送，有几个字段是必需的：

1. sender_id：发送账号的id/tag
2. receivers：接收账号的id/tag
3. message_type：如今的IM都支持多媒体，所以使用该字段标识消息类型
4. message_data：对应message_type的消息体

以及，设计两个基础字段：

1. id：用于进行一定的标识，从而可以实现基于id获取配置的功能。往往IM的配置相对静态，可以通过id获取IM的相关配置，减少在metadata中的字段
2. saas_name：多IM场景时使用，保留这样的可拓展性，但一般用不到该场景。当用到该场景时，由于不同IM的发送参数有巨大差异，如果在代码中传递metadata等字段，则可能无法移植，这时候就需要上述的 id 功能。

一个密钥字段：

1. secret：往往是accessToken，但也可通过上述 id 功能，由id获取对应的配置，在配置中获取accessToken。所以该字段往往不需要显式传入，但仍然在API中进行保留，以便于直接调用。

Signed-off-by: bokket [email protected]

What this PR does: add Hdfs oss support for Layotto file system interface.

Which issue(s) this PR fixes:

Fixes #236

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Add Layotto secret API. To generate key dependent interface definitions, similar to dapr

English explanation will be added later

Secret API设计文档

本文档讨论"密钥相关"的API

1. 需求

1.1. 密钥管理

Q: 为何需要密钥管理？

A: 应用程序在执行过程中大多数会使用专门的密钥去访问外部服务，通常需要建议一个对应的密钥存储来管理。

Q: Layotto Secret API充当的角色？

A: 提供管理密钥以及允许开发人员通过API获得密钥的功能，充当密钥管理者角色。

2. 产品调研

| 平台方 | 提供功能 | |--|--| |dapr|访问权限（可依赖各自云产品支持），密钥管理，检索密钥| |云密钥产品相关|访问权限，密钥管理，检索密钥|

3. grpc API设计

在制定对应API时，需要考虑每个密钥的唯一资源定位因素有哪些？

• 存放空间地址
• 密钥对应名称
• 密钥对应版本
• 密钥对应状态

  其中最为主要的则是前两个。版本默认latest，而状态则默认取生效，如没有生效的密钥，则需要业务去判断下一步操作。

3.1. proto定义

  // Gets secrets from secret stores.
  rpc GetSecret(GetSecretRequest) returns (GetSecretResponse) {}

  // Gets a bulk of secrets
  rpc GetBulkSecret(GetBulkSecretRequest) returns (GetBulkSecretResponse) {}
  
  // GetSecretRequest is the message to get secret from secret store.   
message GetSecretRequest {
  // The name of secret store.
  string store_name = 1;

  // The name of secret key.
  string key = 2;

  // The metadata which will be sent to secret store components.
  // Contains version, status, and so on...
  map<string,string> metadata = 3;
}

// GetSecretResponse is the response message to convey the requested secret.
message GetSecretResponse {
  // data is the secret value. Some secret store, such as kubernetes secret
  // store, can save multiple secrets for single secret key.
  map<string, string> data = 1;
}


// GetBulkSecretRequest is the message to get the secrets from secret store.
message GetBulkSecretRequest {
  // The name of secret store.
  string store_name = 1;

  // The metadata which will be sent to secret store components.
  map<string,string> metadata = 2;
}


// GetBulkSecretResponse is the response message to convey the requested secrets.
message GetBulkSecretResponse {
  // data hold the secret values. Some secret store, such as kubernetes secret
  // store, can save multiple secrets for single secret key.
  map<string, SecretResponse> data = 1;
}


// SecretResponse is a map of decrypted string/string values
message SecretResponse {
  map<string, string> secrets = 1;
}

Q: 是否由Layotto帮用户实现对应密钥获取实现？

A: API和Layotto运行时不管这事，由sdk或者用户自己处理，或者某个特殊组件想实现这个feature也可以。

4. 组件API

package secretstores

type SecretStore interface {
		// Init authenticates with the actual secret store and performs other init operation
	Init(metadata Metadata) error
	// GetSecret retrieves a secret using a key and returns a map of decrypted string/string values
	GetSecret(req GetSecretRequest) (GetSecretResponse, error)
	// BulkGetSecrets retrieves all secrets in the store and returns a map of decrypted string/string values
	BulkGetSecret(req BulkGetSecretRequest) (BulkGetSecretResponse, error)
}

// GetSecretRequest describes a get secret request from a secret store.
type GetSecretRequest struct {
	Name     string            `json:"name"`
	Metadata map[string]string `json:"metadata"`
}

// BulkGetSecretRequest describes a bulk get secret request from a secret store.
type BulkGetSecretRequest struct {
	Metadata map[string]string `json:"metadata"`
}

// GetSecretResponse describes the response object for a secret returned from a secret store.
type GetSecretResponse struct {
	Data map[string]string `json:"data"`
}

// BulkGetSecretResponse describes the response object for all the secrets returned from a secret store.
type BulkGetSecretResponse struct {
	Data map[string]map[string]string `json:"data"`
}

// DefaultSecretRefKeyName is the default key if secretKeyRef.key is not given.
const DefaultSecretRefKeyName = "_value"

// Metadata contains a secretstore specific set of metadata properties.
type Metadata struct {
	Properties map[string]string `json:"properties,omitempty"`
}

参考资料

dapr密钥管理概览

AWS Secrets Manager

Microsoft Key Vault

Azure Key Vault

阿里云密钥管理服务

What this PR does:

add LockKeepAlive function for renewal lease

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

What this PR does: reopen for https://github.com/mosn/layotto/pull/820 Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

1. Background

The wit-bindgen project officially provided by WebAssembly can enable better interaction between the wasm module and the runtime, avoiding the development of various data conversion logics, and now Layotto integrates wasmtime, so we want to integrate wit-bindgen to improve the user's ability to develop wasm modules experience.

2. What I have tried

1. define a wit file：

record person {
  name: string,
  age: u32,
}

test: func(who: person) -> string

2. develop user module

use hello::*;

wit_bindgen_rust::import!("wit/hello.wit");

fn main() {
    let p = Person{name:"gj", age: 18};
    let result = test(p);
    print!("result: {}", result);
}

3. add implementation in wasmtime

use hello::*;

wit_bindgen_rust::export!("/Users/gujin/workspace/rust/hello-wasmtime/wit/hello.wit");

struct Hello {}

impl hello::Hello for Hello {
    fn test(p: Person) -> String {
        return p.name;
    }
}

//register to wasmtime

In this way, the test function can be implemented, but how to register this implementation with wasmtime and provide services to users has not yet been found.

3. what need to do

1. Learn how to integrate wit-bindgen with wasmtime
2. Learn how to integrate wit-bindgen with wasmtime-go

References：

1. https://github.com/bytecodealliance/wit-bindgen
2. https://github.com/fermyon/spin
3. discussion：#611

中文

一、背景

WebAssembly 官方提供的 wit-bindgen 项目可以让 wasm 模块跟运行时之间更好的交互，避免开发各种数据转换逻辑，现在Layotto集成了wasmtime，因此想要集成wit-bindgen来提高用户开发wasm模块的体验。

二、做过的一些尝试

1. 自定义一个wit文件：

record person {
  name: string,
  age: u32,
}

test: func(who: person) -> string

2. 用户开发时使用

use hello::*;

wit_bindgen_rust::import!("wit/hello.wit");

fn main() {
    let p = Person{name:"gj", age: 18};
    let result = test(p);
    print!("result: {}", result);
}

3. 运行时提供具体实现

use hello::*;

wit_bindgen_rust::export!("/Users/gujin/workspace/rust/hello-wasmtime/wit/hello.wit");

struct Hello {}

impl hello::Hello for Hello {
    fn test(p: Person) -> String {
        return p.name;
    }
}

//把实现注册到wasmtime

这样可以实现test接口，但是如何把该实现注册到wasmtime，对用户提供服务还没有找到办法。

三、需要做什么

1. 调研用wasmtime集成wit-bindgen的方法
2. 调研用wasmtime-go集成wit-bindgen的方法

参考资料：

1. https://github.com/bytecodealliance/wit-bindgen
2. https://github.com/fermyon/spin
3. 之前的讨论：#611

What would you like to be added:

Develop a java demo for OSS API.

Now we already have quickstart docs for this API, and we need a demo to show users how to invoke this API.

You can take the lock api quickstart as an example:

The java demo of lock api is here.

Reference:

Here are some resources to help you work on this issue.

Quickstart doc for this API: https://mosn.io/layotto/#/en/start/oss/start

API definition (the .proto file): https://github.com/mosn/layotto/blob/main/spec/proto/extension/v1/s3/oss.proto

java sdk: https://github.com/layotto/java-sdk

API design doc/discussion: https://mosn.io/layotto/#/zh/design/oss/design

Why is this needed:

Java users need it.

What would you like to be added:

Develop a new component for delay queue API.

Choose an open source component or cloud service you like to implement this API.

For example, you can choose any one in the list below:

• [ ] RocketMQ
• [ ] Active MQ
• [ ] Rabbit MQ
• [ ] QMQ
• [ ] Cloud service provided by Alibaba cloud
• [ ] Cloud service provided by AWS
• [ ] Cloud service provided by Google cloud
• [ ] Cloud service provided by tencent cloud
• [ ] Cloud service provided by qiniu cloud
• [ ] anything else

What is this API used for?: DelayQueue is a special kind of message queue, which lets you postpone the delivery of new messages to consumers. For example, you can invoke this API and tell the message queue "please send this message to the consumers after 5 minutes".

Reference:

Here are some resources to help you work on this issue.

API definition(the .proto file): https://github.com/mosn/layotto/blob/main/spec/proto/extension/v1/delay_queue/delay_queue.proto

Component development guide: https://mosn.io/layotto/#/en/development/developing-component

Component development guide(in chinese): https://mosn.io/layotto/#/zh/development/developing-component

An example pull request to develop a new component: #807

API design doc/discussion: https://github.com/mosn/layotto/issues/617

chinese: 选择一个你喜欢的开源组件或云服务实现延迟消息 API, 参考资料见上。

Why is this needed:

We need more components to make this API useful

What would you like to be added:

Develop a new component for email API.

Choose an open source component or cloud service you like to implement this API.

For example, you can choose any one in the list below:

• [ ] Cloud service provided by Alibaba cloud
• [ ] Cloud service provided by AWS
• [ ] Cloud service provided by Azure
• [ ] Cloud service provided by Google cloud
• [ ] Cloud service provided by tencent cloud
• [ ] Cloud service provided by qiniu cloud
• [ ] anything else

Reference:

An example pull request :

API quickstart doc:

API reference: https://mosn.io/layotto/api/v1/email.html

API design doc/discussion: https://github.com/mosn/layotto/issues/717

chinese: 选择一个你喜欢的开源组件或云服务实现 email API 参考资料见上。

Why is this needed:

We need more components to make this API useful