Orion - a small lightweight framework written around grpc/protobuf with the aim to shorten time to build microservices at Carousell.

Last update: Nov 16, 2022

Comments: 16

Orion

Orion is a small lightweight framework written around grpc/protobuf with the aim to shorten time to build microservices at Carousell.

It is derived from 'Framework' a small microservices framework written and used inside https://carousell.com, It comes with a number of sensible defaults such as zipkin tracing, hystrix, live reload of configuration, etc.

Getting Started

Follow the guide at https://github.com/carousell/Orion/blob/master/orion/README.md

Setup Instructions

Orion is written in golang, please follow instructions on https://golang.org/doc/install to install, or you can also run

brew install golang

sudo dnf install golang

Make sure you are on go 1.9 or later add the following lines to your ~/.profile

export GOPATH="$HOME/code/go"
export GOBIN="$GOPATH/bin"
export PATH="$GOBIN:$PATH"
export PATH="$HOME/.gotools:$PATH"

source your ~/.profile

source ~/.profile

then create the code dir

mkdir -p $GOPATH

we use govendor to vendor package in Orion, install it by running

go get -u github.com/kardianos/govendor

another helpful tool to check for unupdated packages is Go-Package-Store, install it by running

go get -u github.com/shurcooL/Go-Package-Store/cmd/Go-Package-Store

now clone this repo

mkdir -p $GOPATH/src/github.com/carousell/
git clone [email protected]:carousell/Orion.git $GOPATH/src/github.com/carousell/Orion

You need the following tools to better develop for go

go get -u golang.org/x/lint/golint

now you can build the package by using make build

gRPC

for gRPC, you need to follow the following steps

get gRPC codebase

go get -u google.golang.org/grpc

install protobuf

brew install protobuf

install the protoc plugin for go

go get -u github.com/golang/protobuf/{proto,protoc-gen-go}

install the protoc plugin for orion

go get -u github.com/carousell/Orion/protoc-gen-orion

Project Status

Orion is in use at production at Carousell and powers multiple (100+) services serving thousands of requests per second, we ensure all updates are backward compatible unless it involves a major bug or security issue.

License

This code is available under the following https://github.com/carousell/Orion/blob/master/LICENSE

Owner

Carousell

Carousell is a community marketplace that makes selling as simple as taking a photo, buying as simple as chatting.

https://github.com/carousell/Orion

Comments

[BSX-265] improve grpc-logging

https://carousell.atlassian.net/browse/BSX-265

old log format

{"@timestamp":"2022-12-06T08:43:44.955187131Z","caller":"interceptors/unary_interceptors.go:38","error":null,"grpcMethod":"/warrantycorepb.WarrantyCoreService/GetWarrantyOptionGroup","level":"info","method":"/warrantycorepb.WarrantyCoreService/GetWarrantyOptionGroup","took":"7.902368ms","trace":"1802ae6a-7542-11ed-926a-8ac647f8a923","trace_id":"1d4a11bb0c05c2ba","transport":"gRPC"}

new log format

{"@timestamp":"2022-12-06T08:34:51.817602277Z","caller":"interceptors/unary_interceptors.go:74","grpcMethod":"/warrantycorepb.WarrantyCoreService/GetWarrantyOptionGroup","level":"info","req":"{\"ccid\":\"2799\",\"listing_type\":[1],\"warranty_option_length_unit\":1}","resp":"{\"warranty_option_group\":{\"ccid\":\"2799\",\"options\":[{\"id\":\"a85fef11-1ff9-46d6-ad63-ce76735c7bf4\",\"currency\":\"MYR\",\"price\":\"0\",\"length\":1},{\"id\":\"e63ae5d0-848c-4f3a-bca5-df26152d328e\",\"currency\":\"MYR\",\"price\":\"49\",\"length\":6},{\"id\":\"8f1aa172-d126-4060-8665-4f7d6ed84347\",\"currency\":\"MYR\",\"price\":\"88\",\"length\":12,\"is_best\":true},{\"id\":\"f0c319dc-2a82-47b6-a2ab-8648cb93d6ea\",\"currency\":\"MYR\",\"price\":\"180\",\"length\":24}],\"best_option_id\":\"8f1aa172-d126-4060-8665-4f7d6ed84347\"}}","took":"4.534181ms","trace":"da3ccd00-7540-11ed-a081-aaa3ee5b0697","trace_id":"79d71ec1fa946f59","transport":"gRPC"}

What we have changed in this PR:

add the logging for req and resp, default not print
remove to print "error":null if no error
remove to print method as it is already print in grpcMethod

Config to disable default interceptors
This PR is allow service to disable Orion's default interceptors by setting config

[orion] DisableDefaultInterceptors=true

This could be useful to disable interceptors such as ResponseTimeLoggingInterceptor which logs massively for each endpoint calls.

Fix Potential Race on Logger

Issue Description When multiple parallel routine try to log like log.GetLogger().Log(...), there is a potential race condition in logger.GetLogger.

==================
WARNING: DATA RACE
Write at 0x0000014994b0 by goroutine 9:
  github.com/carousell/Orion/utils/log.GetLogger.func1()
      /.../Orion/utils/log/log.go:65 +0xaa
  sync.(*Once).Do()
      /usr/local/go/src/sync/once.go:44 +0xde
  github.com/carousell/Orion/utils/log.GetLogger()
      /.../Orion/utils/log/log.go:64 +0xaf
  github.com/carousell/Orion.main.func1()
      /.../Orion/racecheck.go:22 +0x187

Previous read at 0x0000014994b0 by goroutine 8:
  github.com/carousell/Orion/utils/log.GetLogger()
      /.../Orion/utils/log/log.go:63 +0x3e
  github.com/carousell/Orion.main.func1()
      /.../Orion/racecheck.go:22 +0x187

Goroutine 9 (running) created at:
  github.com/carousell/Orion.main()
      /.../Orion/racecheck.go:18 +0xb5
  github.com/carousell/Orion.Test_main.func1()
      /.../Orion/racecheck_test.go:18 +0x2f
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:865 +0x163

Goroutine 8 (running) created at:
  github.com/carousell/Orion.main()
      /.../Orion/racecheck.go:18 +0xb5
  github.com/carousell/Orion.Test_main.func1()
      /.../Orion/racecheck_test.go:18 +0x2f
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:865 +0x163
==================

Test Code

go test -v -race -run main

racecheck_test.go a empty test just for go test race check

package main

import "testing"

func Test_main(t *testing.T) {

        // TODO: Init instances and mocks

        testCases := []struct {
                name string
        }{
                {
                        name: "hi",
                },
        }
        for _, tt := range testCases {
                t.Run(tt.name, func(t *testing.T) {
                        main()
                })
        }
}

racecheck.go simulate parallel GetLogger() case

package main

import (
        "context"
        "fmt"
        "sync"

        "github.com/carousell/Orion/utils/errors"
        "github.com/carousell/Orion/utils/log"
        "github.com/carousell/Orion/utils/log/loggers"
)

func main() {
        var wg sync.WaitGroup

        for i := 0; i < 2; i++ {
                wg.Add(1)
                go func(wg *sync.WaitGroup, routineIndex int) {
                        defer wg.Done()
                        errWithStack := errors.WrapWithSkip(fmt.Errorf("error %d", routineIndex), "", 1)
                        log.GetLogger().Log(context.Background(), loggers.ErrorLevel, 1, "err", errWithStack, "stack", errWithStack.StackFrame())
                }(&wg, i)
        }
        wg.Wait()
}

Support tool installation after Go 1.17

Situation:

Starting in Go 1.17, installing executables with go get is deprecated. go install may be used instead. And go get is no longer supported outside a module in Go 1.18. (Source)

Which means Orion is no longer available in Go 1.18. We will stuck with older version of Go in the future.

Install protoc-gen-orion with different Go version.

||Go 1.16|Go 1.17|Go 1.18| |-|-|-|-| |go get|Passed|Warned|Failed| |go install|Failed|Failed|Failed|

go get -u github.com/carousell/Orion/protoc-gen-orion go install github.com/carousell/Orion/protoc-gen-orion@latest

Target:

Allow protoc-gen-orion to be installed by go install.

Action:

Remove replace directives from go.mod.

Result:

We can install protoc-gen-go with go install command in Go 1.16, 1.17, and 1.18.

Install protoc-gen-orion with different Go version.

||Go 1.16|Go 1.17|Go 1.18| |-|-|-|-| |go get|Passed|Warned|Failed| |go install|Passed|Passed|Passed|

go get -u github.com/carousell/Orion/protoc-gen-orion@afd0407a go install github.com/carousell/Orion/protoc-gen-orion@afd0407a afd0407a = this commit
init incoming context as default for HTTP server

We've onboarded forward metadata via incoming and outgoing context in https://github.com/carousell/Orion/pull/149, but in HTTP server we haven't inited an incoming context inside, which means we can use the metadata in the incoming context that will be transformed to outgoing when we call to other services.

Also, in our current interface of encoders: https://github.com/carousell/Orion/blob/master/orion/handlers/types.go#L40-L44, it doesn't support context assigning back, which will cause new context with incoming context lost. We will have to init an incoming context in preparation stage. (Btw, for gRPC server, the context always has incoming context regardless)
Add GRPCRecovery Interceptor
main purpose is to prevent silent crash because of server panic, should recover and send the error out (through notifier/log).

add GRPCRecoveryInterceptor to DefaultInterceptors()

in GRPCRecoveryInterceptor, it will use grpc_recovery middleware with defined handler return an Orion/utils/errors ErrorExt. The error will skip 4 stack frames and show the function triggered the panic.

Concept Flow not include the change for http middlewares https://drive.google.com/file/d/13AkM-68szUG_ITRaWJUpnJGcQF4Xqb3J/view?usp=sharing

Panic Log with no skip

{"@timestamp":"2019-07-01T16:19:33.3244498Z","caller":"interceptors/interceptors.go:137","err":"runtime error: index out of range","grpcMethod":"/{service}/{rpcName}","level":"error", "stack":[ {"file":"/go/src/github.com/.../vendor/github.com/carousell/Orion/interceptors/interceptors.go","line":153,"function":"GRPCRecoveryInterceptor.func1.1"}, {"file":"/usr/local/go/src/runtime/asm_amd64.s","line":522,"function":"call32"}, {"file":"/usr/local/go/src/runtime/panic.go","line":513,"function":"gopanic"}, {"file":"/usr/local/go/src/runtime/panic.go","line":44,"function":"panicindex"}, {"file":"/go/src/github.com/.../service.go","line":442,"function":"{panic func}"}, ...

so skip 4 stack frames to remove non-related information.

Example on Sentry (Internal) https://sentry.carou.sl/carousell/home/issues/244262/?query=is:unresolved
Allow *.orion.pb.go files to access the exported-service-desc from *.pb.go
Allow *.orion.pb.go files to access the exported-service-desc from *.pb.go

Add two execution parameters.

standalone-mode: Allow you to place *.orion.pb.go and *.pb.go into different folder.

exported-service-desc: Allow you to use the higher version (v1.20.0) of protoc-gen-go.

Read go_package option from protobufs files. In standalone mode, the external import path is getting from the go_package option.

https://github.com/carousell/Orion/issues/186
reorder defaultclientinterceptors

A request that failed could be many reasons including network-level issues, connection re-establishment, etc.. Hystrix as implementing circuit breaker pattern considers how to avoid intermittent network hiccups as well. Also, we don't want to have circuit open because of too many retries that were probably intended.

In other words, we consider Hystrix Command as a whole for a request to other services regardless of whether retry or not.

However, it may be a breaking change to some services that have been using grpc_retry options. We will have to notice those services after we merge this PR.
Add version config to kafka utils
Problem

In a recent exercise to onboard trace-id in logging, we upgraded Orion to the latest commit. However, due to complex dependencies between several internal modules, we are forced to upgrade sarama to v1.30.0 (the dependent version by carousell/messaging).

With this upgrade, it breaks out publisher to old v0.11.2 kafka cluster. Connecting to kafka client also failed with error error creating async producer: kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

Root Cause

The root cause is since sarama v1.27.1, the default kafka version in config is changed from v0.8.2.0 to v1.0.0.0.

If you are connecting to old kafka cluster using v1.27.1+, you have to explicitly specify the Version in sarama config https://github.com/Shopify/sarama/blob/v1.27.1/config.go#L425-L431.

PR Change Summary

However, Orion's NewProducer() doesn't expose an option for callers to set version config.

This PR introduces a new version option allowing callers to specify kafka version when connecting to old version cluster.

Usage

import ( "github.com/Shopify/sarama" "github.com/carousell/Orion/utils/kafka" ) producer, err := kafka.NewProducer( []string{"broker1", "broker2", "broker3"} kafka.WithVersion(sarama.V0_11_0_2), // other options go here... )
Adding sonar.properties and .golangci.yml template files
Adding sonarqube properties files as a part of service creation

Adding .golangci.yml as a part of service creation

Adding phase to add golangci dependency
Fix travis ci error

We have an error when travis building code on tip env with go1.19. Run go mod tidy for download missing packages on the env. Add more go version for building the code.
Allow *.orion.pb.go files to be generated into a different package from the generated protobufs
Currently the generated *.orion.pb.go file needs to be placed as a sibling to the generated protobufs, because the orion file references an unexported variable:

https://github.com/carousell/Orion/blob/ec021a79aefef548f78fd349884cb3f50b8db9ed/protoc-gen-orion/orion.go#L262

In later versions of protoc-gen-go-grpc, this variable is now exported. https://github.com/carousell/Orion/issues/171 is a related issue to allow protoc-gen-orion to use the exported variable.

The variable being exported now also means that it should be possible to place the *.orion.pb.go file in a separate package. In Carousell we're considering having a Go module for all generated protobufs, and another module for all generated orion *.orion.pb.go files.

Ideally the module for generated protobufs will only contain two dependencies:

github.com/golang/protobuf google.golang.org/grpc

Having the orion files as a sibling will require github.com/carousell/Orion to be added as another dependency. The orion dependency is too heavy for us to include. Not every service that consumes our generated protobufs will need Orion.

Thus ideally the orion file will live in a separate package. Example file structure:

gen/ go/ serviceA/serviceA.pb.go serviceB/serviceB.pb.go go.mod orion/ serviceA/serviceB.orion.pb.go go.mod

I have some ideas of how to extend protoc-gen-orion to support this. I'll add more comments to the issue later.

[RE-5863] Resolve potential race condition in log fields

Description

In current version of logFields, we discovered that there is a potential risk of having racing condition when multiple goroutine try to access logFields under an identical context (ctx). As you may see from the report below, when running parallelism test on the current implementation, a race condition is emerged even with a very little sample rate. Therefore, in this PR, we are turning the old logFields into protected logFields by applying read/write lock. Now when logFields being accessed, it will first check if some other instance is also accessing it, if yes, it will wait until the access is freed up so no simultaneous write will happen. In effects, FromContext doesn't seems to have it's performance decreased, while AddToLogContext increased about 10% of it's process time, and it's all being said with race condition is no longer a concern.

Review Guideline

FromContext Applied with read lock as it's a public method, now we are return a copy of the map instead of giving access directly to logFields

fromContext Same implementation as before, but now it's only for private access

AddToLogContext Now write access is locked when writing new data to logFields

How to Test

Parallelism Test Previously

Reader 2 read from logfields map[k1:v1 k2:v2]
Reader 3 read from logfields map[k1:v1 k2:v2]
Reader 1 read from logfields map[k1:v1 k2:v2]
==================
WARNING: DATA RACE
Write at 0x00c00009f380 by goroutine 14:
  runtime.mapassign_faststr()
      /Users/chiahewen/.goenv/versions/1.16.8/src/runtime/map_faststr.go:202 +0x0
  github.com/carousell/Orion/utils/log/loggers.LogFields.Add()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:19 +0x11d
  github.com/carousell/Orion/utils/log/loggers.AddToLogContext()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:38 +0xcd
  github.com/carousell/Orion/utils/log/loggers/test_test.writeWorker()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:29 +0x1ce
  github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelWrite.func1()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:61 +0xa4

Previous write at 0x00c00009f380 by goroutine 16:
  runtime.mapassign_faststr()
      /Users/chiahewen/.goenv/versions/1.16.8/src/runtime/map_faststr.go:202 +0x0
  github.com/carousell/Orion/utils/log/loggers.LogFields.Add()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:19 +0x11d
  github.com/carousell/Orion/utils/log/loggers.AddToLogContext()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:38 +0xcd
  github.com/carousell/Orion/utils/log/loggers/test_test.writeWorker()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:29 +0x1ce
  github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelWrite.func1()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:61 +0xa4

Goroutine 14 (running) created at:
  github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelWrite()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:59 +0x204
  testing.tRunner()
      /Users/chiahewen/.goenv/versions/1.16.8/src/testing/testing.go:1193 +0x202

Goroutine 16 (running) created at:
  github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelWrite()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:59 +0x204
  testing.tRunner()
      /Users/chiahewen/.goenv/versions/1.16.8/src/testing/testing.go:1193 +0x202
==================
Writer 3 wrote key3:val8081
Writer 2 wrote key2:val1847
Writer 1 wrote key1:val7887
lf map[key1:val7887 key2:val1847 key3:val8081 test-key:test-value]
--- FAIL: TestParallelWrite (0.25s)
    testing.go:1092: race detected during execution of test
==================
WARNING: DATA RACE
Write at 0x00c000196090 by goroutine 23:
  runtime.mapassign_faststr()
      /Users/chiahewen/.goenv/versions/1.16.8/src/runtime/map_faststr.go:202 +0x0
  github.com/carousell/Orion/utils/log/loggers.LogFields.Add()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:19 +0x11d
  github.com/carousell/Orion/utils/log/loggers.AddToLogContext()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:38 +0xcd
  github.com/carousell/Orion/utils/log/loggers/test_test.writeWorker()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:29 +0x1ce
  github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelReadAndWrite.func2()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:93 +0xa4

Previous write at 0x00c000196090 by goroutine 22:
  runtime.mapassign_faststr()
      /Users/chiahewen/.goenv/versions/1.16.8/src/runtime/map_faststr.go:202 +0x0
  github.com/carousell/Orion/utils/log/loggers.LogFields.Add()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:19 +0x11d
  github.com/carousell/Orion/utils/log/loggers.AddToLogContext()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:38 +0xcd
  github.com/carousell/Orion/utils/log/loggers/test_test.writeWorker()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:29 +0x1ce
  github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelReadAndWrite.func2()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:93 +0xa4

Goroutine 23 (running) created at:
  github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelReadAndWrite()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:91 +0x2c5
  testing.tRunner()
      /Users/chiahewen/.goenv/versions/1.16.8/src/testing/testing.go:1193 +0x202

Goroutine 22 (running) created at:
  github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelReadAndWrite()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:91 +0x2c5
  testing.tRunner()
      /Users/chiahewen/.goenv/versions/1.16.8/src/testing/testing.go:1193 +0x202
==================
Writer 3 wrote key3:val1318
Reader 3 read from logfields map[key1:val4059 key2:val2081 key3:val1318 test-key:test-value]
==================
WARNING: DATA RACE
Read at 0x00c00019c098 by goroutine 20:
  reflect.typedmemmove()
      /Users/chiahewen/.goenv/versions/1.16.8/src/runtime/mbarrier.go:177 +0x0
  reflect.copyVal()
      /Users/chiahewen/.goenv/versions/1.16.8/src/reflect/value.go:1310 +0x7d
  reflect.(*MapIter).Value()
      /Users/chiahewen/.goenv/versions/1.16.8/src/reflect/value.go:1264 +0x186
  internal/fmtsort.Sort()
      /Users/chiahewen/.goenv/versions/1.16.8/src/internal/fmtsort/sort.go:65 +0x2d9
  fmt.(*pp).printValue()
      /Users/chiahewen/.goenv/versions/1.16.8/src/fmt/print.go:769 +0x14cf
  fmt.(*pp).printArg()
      /Users/chiahewen/.goenv/versions/1.16.8/src/fmt/print.go:712 +0x284
  fmt.(*pp).doPrintf()
      /Users/chiahewen/.goenv/versions/1.16.8/src/fmt/print.go:1026 +0x330
  fmt.Fprintf()
      /Users/chiahewen/.goenv/versions/1.16.8/src/fmt/print.go:204 +0x84
  fmt.Printf()
      /Users/chiahewen/.goenv/versions/1.16.8/src/fmt/print.go:213 +0x111
  github.com/carousell/Orion/utils/log/loggers/test_test.readWorker()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:23 +0x67
  github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelReadAndWrite.func1()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:85 +0x99

Previous write at 0x00c00019c098 by goroutine 22:
  github.com/carousell/Orion/utils/log/loggers.LogFields.Add()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:19 +0x132
  github.com/carousell/Orion/utils/log/loggers.AddToLogContext()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/fields.go:38 +0xcd
  github.com/carousell/Orion/utils/log/loggers/test_test.writeWorker()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:29 +0x1ce
  github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelReadAndWrite.func2()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:93 +0xa4

Goroutine 20 (running) created at:
  github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelReadAndWrite()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:83 +0x20e
  testing.tRunner()
      /Users/chiahewen/.goenv/versions/1.16.8/src/testing/testing.go:1193 +0x202

Goroutine 22 (running) created at:
  github.com/carousell/Orion/utils/log/loggers/test_test.TestParallelReadAndWrite()
      /Users/chiahewen/go/1.16.8/src/github.com/carousell/Orion/utils/log/loggers/test/parallelism_test.go:91 +0x2c5
  testing.tRunner()
      /Users/chiahewen/.goenv/versions/1.16.8/src/testing/testing.go:1193 +0x202
==================
Reader 2 read from logfields map[key1:val4059 key2:val2081 key3:val1318 test-key:test-value]
Reader 1 read from logfields map[key1:val4059 key2:val2081 key3:val1318 test-key:test-value]
Writer 1 wrote key1:val4059
Writer 2 wrote key2:val2081
map[key1:val4059 key2:val2081 key3:val1318 test-key:test-value]
--- FAIL: TestParallelReadAndWrite (0.25s)
    testing.go:1092: race detected during execution of test
FAIL
exit status 1
FAIL    github.com/carousell/Orion/utils/log/loggers/test       1.136s

Now

Reader 9 read from logfields map[k1:v1 k2:v2]
Reader 2 read from logfields map[k1:v1 k2:v2]
...
Writer 47 wrote key47:val4078
Writer 48 wrote key48:val6159
lf map[key1:val1847 key10:val8511 key11:val3300 key12:val694 key13:val1211 key14:val8162 key15:val3237 key16:val495 key17:val9947 key18:val4728 key19:val3274 key2:val7887 key20:val5089 key21:val1445 key22:val9106 key23:val1528 key24:val5466 key25:val6258 key26:val8047 key27:val8287 key28:val2790 key29:val2888 key3:val1318 key30:val3015 key31:val5541 key32:val408 key33:val7387 key34:val6831 key35:val5429 key36:val1737 key37:val631 key38:val1485 key39:val5194 key4:val8081 key40:val5356 key41:val6413 key42:val3090 key43:val5026 key44:val4147 key45:val563 key46:val4324 key47:val4078 key48:val6159 key49:val2433 key5:val2081 key50:val1353 key6:val4059 key7:val2540 key8:val4425 key9:val456 test-key:test-value]
Reader 29 read from logfields map[test-key:test-value]
Reader 18 read from logfields map[test-key:test-value]
...
map[key1:val1957 key10:val9355 key11:val2451 key12:val4538 key13:val8266 key14:val156 key15:val2605 key16:val5561 key17:val7202 key18:val9828 key19:val4783 key2:val3721 key20:val4376 key21:val5746 key22:val1563 key23:val9002 key24:val9718 key25:val5447 key26:val5094 key27:val1577 key28:val7463 key29:val7996 key3:val3000 key30:val6420 key31:val8623 key32:val953 key33:val1137 key34:val3133 key35:val9241 key36:val59 key37:val3033 key38:val8643 key39:val3891 key4:val7189 key40:val2002 key41:val8878 key42:val9336 key43:val2546 key44:val9107 key45:val7940 key46:val6503 key47:val552 key48:val9843 key49:val2205 key5:val2199 key50:val1598 key6:val8705 key7:val9703 key8:val2888 key9:val8510 test-key:test-value]
PASS
ok      github.com/carousell/Orion/utils/log/loggers/test       1.200s

Benchmark stat Previously

cpu: Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz
BenchmarkFromContext
BenchmarkFromContext-8                  41110058                25.81 ns/op
BenchmarkFromAddToLogContext
BenchmarkFromAddToLogContext-8           2947244               409.4 ns/op

Now

cpu: Intel(R) Core(TM) i7-1068NG7 CPU @ 2.30GHz
BenchmarkFromContext
BenchmarkFromContext-8                  39115351                25.86 ns/op
BenchmarkFromAddToLogContext
BenchmarkFromAddToLogContext-8           2701092               442.9 ns/op

Checklist

[ ] Test on local
[ ] Test on stage
[ ] run go mod tidy and go mod vendor
[ ] change module's version back to master

Screenshots (Optional)

Related PRs (Optional)

Impacted Endpoints (Optional)

Encoder and Decoder concept seems opposite
Orion defines func prototypes Encoder and Decoder for handling http request/response as https://github.com/carousell/Orion/blob/master/orion/handlers/types.go#L40-L44:

//Encoder is the function type needed for request encoders type Encoder func(req *http.Request, reqObject interface{}) error //Decoder is the function type needed for response decoders type Decoder func(ctx context.Context, w http.ResponseWriter, encodeError, endpointError error, respObject interface{})

Encoder extracts information from incoming http request and convert to a request object. Decoder takes a response object to write it into http response.

Essentially, Encoders does unmarshalling and deserializing action, while Decoders does marshalling and serializing. The name and what exactly it does seems opposite.

Even in Orion's code, we can find this confusion: https://github.com/carousell/Orion/blob/master/orion/handlers/http/http.go#L176-L188. In the beginning of the section, the comment says "decoder func", but it calls encoders later.

Should them switch names?
Unable to register service to Orion Server
Context

The grpc.ServiceDesc has been exported from protoc-gen-go-grpc v1.1.0. The protoc-gen-orion generates the file that contains an unresolved reference to the unexported name of service desc.

Process

Use the Orion example.

Execute protoc -I. --go-grpc_out=. --go_out=. --orion_out=. simple.proto.

Environment

go version 1.17.6 protoc-gen-go-grpc v1.2.0 protoc-gen-orion latest protoc-gen-go v1.27.1

Expected result

The simple service should be registered in Orion Serve, and simple_proto/simple.proto.orion.pb.go should not contain any error.

Current result

In simple_proto/simple.proto.orion.pb.go, we got a following error.

Unresolved reference '_SimpleService_serviceDesc'

Possible Fix

Follow the protoc-gen-go-grpc naming role in https://github.com/grpc/grpc-go/pull/4035. And modify the generating code. https://github.com/carousell/Orion/blob/ec021a79aefef548f78fd349884cb3f50b8db9ed/protoc-gen-orion/orion.go#L262
Bump go.elastic.co/apm from 1.4.0 to 1.11.0
Bumps go.elastic.co/apm from 1.4.0 to 1.11.0.

Release notes

Sourced from go.elastic.co/apm's releases.

v1.11.0

Fix CVE-2021-22133

Make TRANSACTION_IGNORE_URLS dynamically configurable: #872

v1.10.0

module/apmsql: add tracingDriver.Unwrap method to get underlying driver #849

module/apmgopgv10: add support for github.com/go-pg/pg/v10 #857 (thanks @macnibblet!)

Enable central configuration of "sanitize_field_names" #856

module/apmgrpc: set span destination context #861

v1.9.0

module/apmgoredisv8: introduce new package to support go-redis v8 (#780)

module/apmhttp: introduce httptrace client option (#788)

module/apmsql: add support for database/sql/driver.Validator (#791)

Record sample rate on transactions and spans, propagate through tracestate (#804)

module/apmredigo: change redigo dependency to v1.8.2 (#807)

Deprecate IGNORE_URLS, replace with TRANSACTION_IGNORE_URLS (#811)

Tracer.Close now waits for the transport goroutine to end before returning (#816)

Relax Kubernetes pod UID discovery rules (#819)

Add transaction and span outcome (#820)

Add cloud metadata, configurable with ELASTIC_APM_CLOUD_PROVIDER (#823)

Round ELASTIC_APM_SAMPLING_RATE with 4 digits precision (#828)

module/apmhttp: implement io.ReaderFrom in wrapped http.ResponseWriter (#830)

Fixed Transaction.Discard so that it sets TransactionData to nil (#836)

module/apmsql/pgxv4: add support for pgx driver (#831 -- thanks @Deepak13245!)

module/apmgormv2: add support for gorm.io (GORM v2) (#825 -- thanks @Deepak13245!)

v1.8.0

Add "recording" config option, to dynamically disable event recording (#737)

Enable central configuration of "stack_frames_min_duration" and "stack_trace_limit" (#742)

Implement "CloseIdleConnections" on the Elasticsearch RoundTripper (#750 - thanks @pebrc!)

Fix apmot nil pointer dereference in Tracer.Inject (#763 - thanks @randomswdev!)

v1.7.2

Update cucumber/godog to 0.8.1 (#733)

v1.7.1

Fix segfault on 32-bit architectures (#728)

v1.7.0

Add span.context.destination.* #664

transport: fix Content-Type for pprof data #679

Add "tracestate" propagation #690

Add support for API Key auth #698

module/apmsql: report rows affected #700

v1.6.0

module/apmhttp: add WithClientRequestName option (#609 - thanks @lobotomist!)

module/apmhttp: add WithPanicPropagation function (#611 - thanks @lobotomist!)

module/apmgoredis: add Client.RedisClient (#613 - thanks @wins1908!)

... (truncated)

Changelog

Sourced from go.elastic.co/apm's changelog.

==== 1.11.0 - 2021/02/01

https://github.com/elastic/apm-agent-go/releases/tag/v1.11.0[View release]

Make TRANSACTION_IGNORE_URLS dynamically configurable: {pull}872#872

[[release-notes-1.10.0]] ==== 1.10.0 - 2021/01/20

https://github.com/elastic/apm-agent-go/releases/tag/v1.10.0[View release]

module/apmsql: add tracingDriver.Unwrap method to get underlying driver {pull}#849[#(849)]

module/apmgopgv10: add support for github.com/go-pg/pg/v10 {pull}857[(#857)]

Enable central configuration of "sanitize_field_names" {pull}856[(#856)]

module/apmgrpc: set span destination context {pull}861[(#861)]

[[release-notes-1.9.0]] ==== 1.9.0 - 2020/11/02

https://github.com/elastic/apm-agent-go/releases/tag/v1.9.0[View release]

module/apmgoredisv8: introduce new package to support go-redis v8 {pull}780[#(780)]

module/apmhttp: introduce httptrace client option {pull}788[#(788)]

module/apmsql: add support for database/sql/driver.Validator {pull}791[#(791)]

Record sample rate on transactions and spans, propagate through tracestate {pull}804[#(804)]

module/apmredigo: change redigo dependency to v1.8.2 {pull}807[#(807)]

Deprecate IGNORE_URLS, replace with TRANSACTION_IGNORE_URLS {pull}811[(#811)]

Tracer.Close now waits for the transport goroutine to end before returning {pull}816[#(816)]

Relax Kubernetes pod UID discovery rules {pull}819[#(819)]

Add transaction and span outcome {pull}820[#(820)]

Add cloud metadata, configurable with ELASTIC_APM_CLOUD_PROVIDER {pull}823[#(823)]

Round ELASTIC_APM_SAMPLING_RATE with 4 digits precision {pull}828[#(828)]

module/apmhttp: implement io.ReaderFrom in wrapped http.ResponseWriter {pull}830[#(830)]

Fixed Transaction.Discard so that it sets TransactionData to nil {pull}836[#(836)]

module/apmsql/pgxv4: add support for pgx driver {pull}831[#(831)]

module/apmgormv2: add support for gorm.io (GORM v2) {pull}825[#(825)]

[[release-notes-1.8.0]] ==== 1.8.0 - 2020/05/06

https://github.com/elastic/apm-agent-go/releases/tag/v1.8.0[View release]

Add "recording" config option, to dynamically disable event recording {pull}737[(#737)]

Enable central configuration of "stack_frames_min_duration" and "stack_trace_limit" {pull}742[(#742)]

Implement "CloseIdleConnections" on the Elasticsearch RoundTripper {pull}750[(#750)]

Fix apmot nil pointer dereference in Tracer.Inject {pull}763[(#763)]

[[release-notes-1.7.2]] ==== 1.7.2 - 2020/03/19

... (truncated)

Commits

c5c7e21 Bump version to 1.11.0

dd3e8c5 Tidy up modelwriter (#888)

697d794 Add ignore_urls to central config (#872)

ba58cd6 v1.10.0 (#873)

580656d internal/apmschema: sync new jsonschema (#864)

46c67f3 module/apmgrpc: set span destination context (#861)

18a8126 model: add "classname" to stacktrace frames (#862)

0a73d5b internal/apmlog: fix log levels (#860)

491755f Make log_level centrally configurable (#859)

879c34a Merge pull request #858 from axw/server-url-singular

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

Orion - a small lightweight framework written around grpc/protobuf with the aim to shorten time to build microservices at Carousell.

Orion

Getting Started

Setup Instructions

gRPC

Project Status

License

Owner

Carousell

Comments

[BSX-265] improve grpc-logging

Config to disable default interceptors

Fix Potential Race on Logger

Support tool installation after Go 1.17

Situation:

Target:

Action:

Result:

init incoming context as default for HTTP server

Add GRPCRecovery Interceptor

Allow *.orion.pb.go files to access the exported-service-desc from *.pb.go

reorder defaultclientinterceptors

Add version config to kafka utils

Problem

Root Cause

PR Change Summary

Usage

Adding sonar.properties and .golangci.yml template files

Fix travis ci error

Allow *.orion.pb.go files to be generated into a different package from the generated protobufs

[RE-5863] Resolve potential race condition in log fields

Description

Review Guideline

How to Test

Checklist

Screenshots (Optional)

Related PRs (Optional)

Impacted Endpoints (Optional)

Encoder and Decoder concept seems opposite

Unable to register service to Orion Server

Context

Process

Environment

Expected result

Current result

Possible Fix

Bump go.elastic.co/apm from 1.4.0 to 1.11.0

v1.11.0

v1.10.0

v1.9.0

v1.8.0

v1.7.2

v1.7.1

v1.7.0

v1.6.0

Related tags

Ephemeral One Time/Build-Time gRPC TLS PKI system.

protoc-gen-grpc-gateway-ts is a Typescript client generator for the grpc-gateway project. It generates idiomatic Typescript clients that connect the web frontend and golang backend fronted by grpc-gateway.

Go-grpc-template - A small template for quickly bootstrapping a, developer platform independent gRPC golang application

Our aim is to expand the capabilities of blockchain and make a secure way for transferring NFT between RMRK and MOVR blockchain.

The aim of this project is to publish and archive newsletters to a target email address.

Using Wireshark to decrypt TLS gRPC Client-Server protobuf messages

⚡️Lightweight framework for microservices & web services in golang

A simple RPC framework with protobuf service definitions

Lightweight http response time based load balancer written in Go

A suite of gRPC debugging tools. Like Fiddler/Charles but for gRPC.

grpc-http1: A gRPC via HTTP/1 Enabling Library for Go

Server and client implementation of the grpc go libraries to perform unary, client streaming, server streaming and full duplex RPCs from gRPC go introduction

Go based grpc - grpc gateway micro service example

Simple grpc web and grpc transcoding with Envoy

Go-grpc - This is grpc server for golang.

GRPC - Creating a gRPC service from scratch

Totem - A Go library that can turn a single gRPC stream into bidirectional unary gRPC servers

Grpc-gateway-map-null - gRPC Gateway test using nullable values in map

Todo-app-grpc - Go/GRPC codebase containing RealWorld examples (CRUD, auth, advanced patterns, etc)

Allow .orion.pb.go files to access the exported-service-desc from .pb.go