Description

Fixes the reminder E2E tests. Fixes to the reminder storage solution with partitioning.

Issue reference

Please reference the issue this PR will close: #3984

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• [ ] Code compiles correctly
• [ ] Created/updated tests
• [ ] Unit tests passing
• [ ] End-to-end tests passing
• [ ] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

RELEASE NOTE: Fix logging of conflict when loading actor state store and fixes to actor reminders partitioning.

Description

Fixes sidecar injection instability. Removes job_test E2E on AKS win and linux. Other E2E stability fixes for linux.

Windows E2E is still unstable. See this run, for example: https://github.com/dapr/dapr/actions/runs/696284536

DO NOT RERUN the run above since we want to keep that for post mortem.

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: N/A

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• [ ] Code compiles correctly
• [ ] Created/updated tests
• [ ] Unit tests passing
• [ ] End-to-end tests passing
• [ ] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

Description

Reuse ASB namespace for E2E tests.

Issue reference

Please reference the issue this PR will close: #3647

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• [ ] Code compiles correctly
• [ ] Created/updated tests
• [ ] Unit tests passing
• [ ] End-to-end tests passing
• [ ] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

Description

Enable SDK E2E test on Windows.

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #2695

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• [ ] Code compiles correctly
• [ ] Created/updated tests
• [ ] Unit tests passing
• [ ] End-to-end tests passing
• [ ] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

Description

Please explain the changes you've made

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #[issue number]

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• [ ] Code compiles correctly
• [ ] Created/updated tests
• [ ] Unit tests passing
• [ ] End-to-end tests passing
• [ ] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

Description

Please explain the changes you've made

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #[issue number]

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• [ ] Code compiles correctly
• [ ] Created/updated tests
• [ ] Unit tests passing
• [ ] End-to-end tests passing
• [ ] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

Signed-off-by: Addison Juarez [email protected]

Description

Issue reference

Please reference the issue this PR will close: #[issue number]

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• [ ] Code compiles correctly
• [ ] Created/updated tests
• [ ] Unit tests passing
• [ ] End-to-end tests passing
• [ ] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

Description

enable dapr to connect to a non-local application read APPLICATION_HOST from environment var.

tested standalone mode. added e2e tests for kubernetes.

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #1972

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• [x] Code compiles correctly
• [x] Created/updated tests
• [x] Unit tests passing
• [x] End-to-end tests passing
• [x] Extended the documentation / Created issue in the https://github.com/dapr/docs/issues/893
• [ ] Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

Description

I noticed a good first issue with a closed incomplete PR Thus I created a new PR and referenced the previous PR.

Things to Note

1. https://github.com/dapr/components-contrib/issues/375 If you upsert and delete and upsert the key, a GetRequest will return nothing because Get Request executes a HGETALL first before using GET. HGETALL returns empty value or or none which is set by the other test via HSET.
2. I also converted the values from bytes to string in the gRPC implementation since Redis inserts an string representation of integers and string.

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #924

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• [x] Code compiles correctly
• [x] Created/updated tests
• [x] Unit tests passing
• [x] End-to-end tests passing
• [x] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs/issues/661
• [ ] Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

Description

Created a new ClusterRole, dapr-operator-admin, and bound the dapr-operator ServiceAccount to that role in favor of cluster-admin.

Issue reference

We strive to have all PR being opened based on an issue, where the problem or feature have been discussed prior to implementation.

Please reference the issue this PR will close: #1997

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• [ ] Code compiles correctly
• [ ] Created/updated tests
• [ ] Unit tests passing
• [ ] End-to-end tests passing
• [ ] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

/area runtime

Describe the proposal

NOTE: I haven't had much time to give this problem a lot of thought or to put together this proposal so please excuse any mistakes or oversights but I wanted to get something out there so we can start to move this area forward.

Prototype: https://github.com/jjcollinge/dapr-external-state

What?

This proposal outlines an approach to enable Dapr to integrate with "external" (not compiled into daprd) state store providers using a gRPC API. There is a bigger objective to allow all components to be external/remote and/or written in other lanugages but this proposal only focuses on the state store building block. This proposal should also not be seen as the ONLY way to use external state stores (components) as we will likely want to support WASM/WASI and possibly other interfaces too at a later date.

Why?

Allowing Dapr to use state stores that have not been contributed back to the upstream project (out of tree) or have been written in a language other than Go is really important for adoption as some people may have private state stores they wish to use or might not be comfortable writing in Go. This is a frequently requested feature and is related to the following issues:

• #3513
• #3496
• #1311
• #739
• #3168

How?

gRPC

As mentioned above, this proposal suggests we use gRPC to integrate with external state store providers initially. For details on why gRPC please refer to this thread (TLDR; Go doesn't have dynamic libraries and WASM/WASI is too restricted currently for full plugins). gRPC can be run over tcp or unix domain sockets so we should look to support both for performance sensitive workloads.

In order to provide an external gRPC state store, Dapr needs to define an interface using protobuf that the external store can implement.

In this case, I've simply mapped the existing Store interface that is used to implement state stores over to protobuf (likely needs refinement).

store.proto

// ------------------------------------------------------------
// Copyright (c) Microsoft Corporation and Dapr Contributors.
// Licensed under the MIT License.
// ------------------------------------------------------------

syntax = "proto3";

package components.proto.state.v1;

import "google/protobuf/empty.proto";
import "common.proto";

// TODO: other language packages...
option go_package = "github.com/dapr/components-contrib/state/proto/v1;state";

// Store service provides a gRPC interface for state store components.
service Store {
  rpc Init(MetadataRequest) returns (google.protobuf.Empty) {}
  rpc Features(google.protobuf.Empty) returns (FeaturesResponse) {}
  rpc Delete(DeleteRequest) returns (google.protobuf.Empty) {}
  rpc Get(GetRequest) returns (GetResponse) {}
  rpc Set(SetRequest) returns (google.protobuf.Empty) {}
  rpc Ping(google.protobuf.Empty) returns (google.protobuf.Empty) {}
  rpc BulkDelete(BulkDeleteRequest) returns (google.protobuf.Empty) {}
  rpc BulkGet(BulkGetRequest) returns (BulkGetResponse) {}
  rpc BulkSet(BulkSetRequest) returns (google.protobuf.Empty) {}
}

message MetadataRequest {
  map<string, string> properties = 1;
}

message FeaturesResponse {
  repeated string feature = 1;
}

message GetRequest {
  string key = 1;
  map<string, string> metadata = 2;
  dapr.proto.common.v1.StateOptions.StateConsistency consistency = 3;
}

message GetResponse {
  bytes data = 1;
  dapr.proto.common.v1.Etag etag = 2;
  map<string, string> metadata = 3;
}

message DeleteRequest {
  string key = 1;
  dapr.proto.common.v1.Etag etag = 2;
  map<string, string> metadata = 3;
  dapr.proto.common.v1.StateOptions options = 4;
}

message SetRequest {
  string key = 1;
  bytes value = 2;
  dapr.proto.common.v1.Etag etag = 3;
  map<string, string> metadata = 4;
  dapr.proto.common.v1.StateOptions options = 5;
}

message BulkDeleteRequest {
  repeated DeleteRequest items = 1;
}

message BulkGetRequest {
  repeated GetRequest items = 1;
}

message BulkStateItem {
  string key = 1;
  bytes data = 2;
  dapr.proto.common.v1.Etag etag = 3;
  string error = 4;
  map<string, string> metadata = 5;
}

message BulkGetResponse {
  repeated BulkStateItem items = 1;
  bool got = 2;
}

message BulkSetRequest {
  repeated SetRequest items = 1;
}

This proto definition should be hosted in the dapr/dapr repo along with the generated client/server stubs and be strongly versioned as per the other Dapr protos.

Dapr

Dapr needs a way to use the client stubs to invoke a remote service that satisfies the gRPC API. In order to do this, we can define a new state store components called external (or something gRPC specific like grpc) in dapr/components-contrib. This state store component implements the Dapr Store interface but is simply an adaptor to invoke a remote gRPC API using the generate Go client. This adaptor will need to map the incoming Dapr state store types to the protobuf types and then invoke the remote method and map back the response.

So how do we use this external state store provider and how does Dapr know how to talk to the external gRPC API? Well the store interface has a Init(metadata state.Metadata) error which is typically used to parse metadata and initialize the state store connection. The metadata struct is just a property bag so for our external state store we can just grab the connection details from here. How will the remote state store be configured? the gRPC API could be pre-configured independently - but I think given that we want to mirror the approach of existing state stores as much as possible - we can simply invoke the Init method remotely and pass the native metadata from the Dapr component across (excluding any additional external properties).

This would allow us to have a component definition like:

apiVersion: dapr.io/v1alpha1
kind: Component
metadata:
  name: statestore
  namespace: statestore
spec:
  type: state.external # Use the external state store type.
  version: v1 # Defines the version of the proto the external state store implements.
  metadata:
  - name: externalAddress # This field is required and will be stripped before sending to the external state store.
    value: localhost:9191
  - name: redisHost # These properties stay the same as the concrete state store.
    value: localhost:6379

Where the state store type is external and we provide this additional property externalAddress to inform Dapr where the remote instance is. The rest of the metadata needs to conform to the concrete state store's schema - in this case Redis - as the remote state store will handle this. We will likely be sending "secrets" across to the external service so we'll need to ensure it is trusted and the transport encrypted (see: #security).

With this, the external state store can open a connection to the remote gRPC server and "proxy" the operations. The Store interface currently doesn't have a Close method so there is currently no way to gracefully close the gRPC connection with the external state store.

External State Store

We can implement the gRPC server API in whatever language we want to (as long as it is supports gRPC). The gRPC server can then handle requests however it needs to - including invoking a custom or proprietary state store.

Given that Dapr defines the gRPC interface, I believe Dapr can provide standard server implementations for the most common languages which uses a similar factory pattern to daprd and components contrib. These implementations will map the gRPC data back into a Dapr model and then invoke a Store interface defined in the native language. This way, to add a new state store component in a different language, a contributor can just implement the Store interface (as you would in Go) and plug it into the appropriate server implementation. We could then also support "built-in" state store components contributed in other languages in these servers whilst also supporting people who have private or separate state stores.

Security

As with all Dapr comms - we'll want to ensure that we are leveraging mTLS between Dapr and the external state store where possible. This is where Dapr providing standardized servers might mean we can hook into Sentry to issue the gRPC server trusted certificates.

Performance

Obviously there's going to be a performance impact by having to invoke an external state store service rather than an in memory implementation. Therefore, users must be informed about the trade off they are making when using an external state store. We can try to minimize the performance impact as much as possible by allowing users to use unix domain sockets as well as tcp. Longer term, I expect the maturity of WASI to become a more performant alternative but that'll take time.

Prototype

I have built a prototype based on this proposal which is available here that implemented the external state store in Go and ruses the existing state stores remotely. Beware this was hacked together very quickly so has sharp corners - but hopefully it's useful to demonstrate the proposal.

This is just an initial draft of an approach to see whether this seems viable - I'm sure there are alternatives that are worth discussing so please feel free to give candid feedback before I progress this any further.

Future

Although I have scoped this issue to just the state store build block - I believe this pattern could be used to support other build block APIs. Depending on the progress of WASM (interface types, WASI, etc.) which might provide a better solution, we may get to a point where we have language proxies for the entire Dapr API.

Describe the feature

Add property named metadata into CloudEvent to support transfer custom vairable. Then add all metadata of publish into it and receive it on consume.

https://docs.dapr.io/developing-applications/building-blocks/pubsub/pubsub-cloudevents/

maybe like this :

{
    "topic": "orders",
    "pubsubname": "order_pub_sub",
    "traceid": "00-113ad9c4e42b27583ae98ba698d54255-e3743e35ff56f219-01",
    "tracestate": "",
    "data": {
    "orderId": 1
    },
    "id": "5929aaac-a5e2-4ca1-859c-edfe73f11565",
    "specversion": "1.0",
    "datacontenttype": "application/json; charset=utf-8",
    "source": "checkout",
    "type": "com.dapr.event.sent",
    "time": "2020-09-23T06:23:21Z",
    "traceparent": "00-113ad9c4e42b27583ae98ba698d54255-e3743e35ff56f219-01"
    "metadata": {
        "custom-key": "custom-value"
     }
}

Performance tests are failing most of the time - https://github.com/dapr/dapr/actions/workflows/dapr-perf.yml.

Likely this started after the k6 addition.

In what area(s)?

/area runtime

What version of Dapr?

1.9.5

Expected Behavior

services:
  myservice:
    image: ...
    restart: always

  myservice-dapr:
    image: "daprio/daprd:edge"
    network_mode: "service:myservice"

If myservice crashes and restarted by docker it should has an ability to restore connection to DAPR

Actual Behavior

myservice failed to connect to DAPR sidecar

Steps to Reproduce the Problem

kill myservice by timer

The problem described in this issue

Suggested solution: do not use network_mode: "service:[service-name]" for sidecar container and pass app-endpoint to daprd. DAPR must restore app connection in case of application crash and restart.

Release Note

RELEASE NOTE:

Signed-off-by: Marcos Candeia [email protected]

Description

As the title says.

Issue reference

Please reference the issue this PR will close: #5714

Checklist

Please make sure you've completed the relevant tasks for this PR, out of the following list:

• [x] Code compiles correctly
• [ ] Created/updated tests
• [ ] Unit tests passing
• [ ] End-to-end tests passing
• [ ] Extended the documentation / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Specification has been updated / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]
• [ ] Provided sample for the feature / Created issue in the https://github.com/dapr/docs/ repo: dapr/docs#[issue number]

In what area(s)?

/area runtime

/area operator

/area placement

/area docs

/area test-and-release

Describe the proposal

Currently, our perf tests executes when a slash command is issue in a given PR. The perf-tests uses the dapr-perf workflow which deploys and runs the declared tests from master branch.

Therefore the workflow file is fetched from master, meaning that the local branch changes are not considered when running perf tests, so the newly added local tests are skipped and will run only when the PR gets merged, which makes difficult to properly set the right thresholds for the tests.

This proposal suggests to migrate the test list from dapr-perf to a make-like command (that already exists) and also migrate from environment variables to hardcoded values specified in the test file.

The only difference is that by default the perf test will run with production parameters that could be overwritten by the environment variables - when specified.