https://github.com/triggermesh/triggermesh/runs/4750579570?check_suite_focus=true

go build ./cmd/...

[...]

# github.com/ibm-messaging/mq-golang/v5/ibmmq
Error: ../../../go/pkg/mod/github.com/ibm-messaging/mq-golang/[email protected]/ibmmq/mqi.go:54:10: fatal error: cmqc.h: No such file or directory
 54 | #include <cmqc.h>
    |          ^~~~~~~~
compilation terminated.

Maybe we need to install extra libraries as part of the workflow.

cc. @tzununbekov

XML to JSON transformation was worked on in a hack-ish way at InfraJS (being deprecated).

https://github.com/triggermesh/triggermesh/blob/22c6a5877a3934f802623848f89be4d057790953/pkg/targets/adapter/infratarget/vm/javascript/javascript_vm.go#L94-L102

XML cannot match one to one with JSON, we should give users the chance to configure how attributes are rendered into JSON and provide documentation on it. For reference, InfraJS does not allow configuring attributes rendering, but documents how it works:

https://docs.triggermesh.io/cloud/targets/infrajs/#xml-support

Moving the logic from the instantiation of the adapter scope, to the application, puts everything inside the CE http request scope and fixes the issue (as far as my testing can tell)...going to perform even more heavy duty testing on it to be sure...opening PR for suggestions / thoughts

issue: https://github.com/triggermesh/triggermesh/issues/1133

The design looks very close to Google Pub/Sub, therefore the source should:

• Take the resource ID of a topic to subscribe to.
• Reconcile a subscription for pulling messages from that topic.

https://docs.microsoft.com/en-us/azure/service-bus-messaging/service-bus-messaging-overview

I'm trying to test out the AWSSQSSource with localstack. For things to work, I think I need to be able to set the AWS endpoint url to something like http://localhost:4566. Overriding the endpoint is possible with, for example, the AWS CLI:

aws --endpoint-url=http://localhost:4566

It seems like it's also possible with the Go AWS SDK, using an endpoint resolver function: https://aws.github.io/aws-sdk-go-v2/docs/configuring-sdk/endpoints/

However, it seems that overriding the AWS endpoint is not possible with a AWSSQSSource. The only related exposed configuration is the arn, which is used to configure the region on the client: https://github.com/triggermesh/triggermesh/blob/dcce7c27b6901bcfb8a3a34720605deec51a570d/pkg/sources/adapter/awssqssource/adapter.go#L133 . Then I think the default endpoint is used based on the service (SQS) and the region.

This PR would allow the ability to passthrough a GUID from the original payload headers, to the inner cloud events payload that is injected into the eventing system.

Why?

The cloud event ID can be an important attribute to mark the start of a transaction (series of events across a system)...by default a unique ID is created automatically. However, if you wanted to use that ID to track the original event -> to the end of the link you would need to know that ID ahead of time.

Since the webhooksource is a resource for taking base JSON payloads and -> to cloud event form...and you can pass attributes, it would make sense I think that you would also be able to pass your own cloud event ID.

Thoughts?

Migrate the Azure source end to end tests to the triggermesh/triggermesh repo. These tests require an azure service principal to be created with the following roles assigned at the subscription level:

• Azure Event Hubs Data Owner
• Azure Service Bus Data owner
• EventGrid Contributor
• Storage Blob Data Owner

Additionally, when running the ginkgo tests, the following environment variables must be defined:

• AZURE_SUBSCRIPTION_ID
• AZURE_TENANT_ID
• AZURE_CLIENT_ID
• AZURE_CLIENT_SECRET
• AZURE_REGION is optional and defaults to uswest2 when omitted

Lastly, this PR updates the makefile to add the tests directory to linting

topic in azureservicebustopicsources is defined on form: topicID: /subscriptions/<subId>/resourceGroups/<rgName>/providers/Microsoft.ServiceBus/namespaces/<someNamespace>/topics/<someTopic> i.e. no option is exposed to tell if adapter should use amqp or wss (websockets). SDK supports wss: https://github.com/Azure/azure-sdk-for-go/commit/2db8a3b9837267a02739eac7962be31a9a1d63f2

Thus; CR should expose option (through a protocol field, or through url-style in existing topicID field) to define if protocol should be amqp, or over websockets.

Websockets are enterprise-friendly where http-proxies are used, and where protocol choice is limited.

Following documentation for AWSS3Source from knative here: https://knative.dev/docs/eventing/sources/#third-party-sources, example deployment https://github.com/triggermesh/triggermesh/blob/main/config/samples/sources/awss3source.yaml goes into CrashLoopBackOff due to small resource limit. But overriding resource limits/requests is not updating the awss3source-sample deployment.

I'm using triggermesh v1.17.0

apiVersion: sources.triggermesh.io/v1alpha1
kind: AWSS3Source
metadata:
  name: sample
spec:
  arn: arn:aws:s3:us-east-1:xxx:yyy

  eventTypes:
  - s3:ObjectCreated:*
  - s3:ObjectRemoved:*

  auth:
    credentials:
      accessKeyID:
        valueFromSecret:
          name: awscreds
          key: AWS_ACCESS_KEY_ID
      secretAccessKey:
        valueFromSecret:
          name: awscreds
          key: AWS_SECRET_ACCESS_KEY

  sink:
    ref:
      apiVersion: serving.knative.dev/v1
      kind: Service
      name: event-display

  adapterOverrides:
    resources:
      limits:
        cpu: "1"
        memory: 256Mi
      requests:
        cpu: 90m
        memory: 30Mi

$ kubectl get deploy awss3source-sample -o yaml|grep -A6 resources:
        resources:
          limits:
            cpu: "1"
            memory: 45Mi
          requests:
            cpu: 90m
            memory: 30Mi

$ kubectl get awss3source
NAME     READY   REASON               QUEUE                                                        SINK                                               AGE
sample   False   AdapterUnavailable   arn:aws:sqs:us-east-1:xxx:zzzz   http://event-display.s3-events.svc.cluster.local   51m

Any help to fix the resource limits is highly appreciated.

edit: formatting (@antoineco)

Will need a commit hook/github action to verify the copyright header corresponds to what is expected for the project including ensuring the copyright year is updated.

This PR introduces the Kind XMLToJSONTransformation It can be sent a Cloudevent containing XML data in the payload and will emit an event containing a JSON representation of said data.

Example Cloudevent:

curl --location --request POST 'http://localhost:8080' \
--header 'Ce-Specversion: 1.0' \
--header 'Ce-Type: test' \
--header 'Ce-Id: 123123' \
--header 'Ce-Source: ser' \
--header 'Content-Type: application/xml' \
--data-raw '<note><to>Tove</to><from>Jani</from><heading>Reminder</heading><body>Dont forget me this weekend</body></note>'

Example Response Payload:

{"note": {"from": "Jani", "heading": "Reminder", "body": "Dont forget me this weekend", "to": "Tove"}}

Bumps github.com/tektoncd/pipeline from 0.40.2 to 0.43.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

We have an org that does not allow iam user access. We are allowed to access objects via the "assume role" https://docs.aws.amazon.com/sdk-for-go/api/aws/credentials/stscreds/

While the following maybe a solution, it is not good enough, as this is only for EKS instances and not for instances outside of EKS. https://github.com/triggermesh/triggermesh/pull/1243

In testing the Redis broker, we found that it attempts to expose port 80 on the container even when the spec.broker.port which controls the service port is set to a different port. This may cause an issue in clusters where privileged ports (under 1024) are not bind-able with unprivileged users.

In our cluster the broker port enters a crash loop with the following error: unable to start HTTP server: error while opening the inbound connection: listen tcp :80: bind: permission denied

This issue was locally resolved by manually editing the security context of the pod and adding the following block to set port 80 as unprivileged:

securityContext:
  sysctls:
  - name: net.ipv4.ip_unprivileged_port_start
    value: "80"

This is officially considered as a safe namespaced sysctl since 1.22.

This has also been raised to Kubernetes to set as a potential default- https://github.com/kubernetes/kubernetes/issues/102612

Could the container port be changed to reflect the port selected in the CRD spec.broker.port so that we can control the exposed port on the pod?

If not possible, could this be added to the pod templates for all resources that attempt to bind privileged ports as for security reasons we may not want to generally enable this within our clusters?

Signed-off-by: Noah Kreiger [email protected]

This change proposes a switch to the logic in the tranformation, for handling a value that does NOT exist.

In a normal coding / average coding flow, you would not usually append a nil value to a path, or the value would not default to the variable name. I find this to be both confusing, and cause issues in practice where you may want to perform routing based on whether a value exists or not, without defining an extra filter, and 2 transformations, etc.

I didn't see any unit tests for this file?

Scenarios:

POST with

{
    "detail": "noah",
    "source": "kreiger"
}

store -> $detail as $detail and $source as $source

add -> $detail.$source -> results in noah.kreiger

POST with

{
    "detail": "noah"
}

store -> $detail as $detail and $source as $source

add -> $detail.$source -> results in noah

POST with

{
    "detail": "noah",
    "source": ""
}

store -> $detail as $detail and $source as $source

add -> $detail.$source -> results in noah.

(still unsure if this should use the empty string, or treat it like a nil)

Bumps golang.org/x/net from 0.2.0 to 0.4.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Bumps github.com/Azure/azure-sdk-for-go/sdk/azidentity from 1.1.0 to 1.2.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.