Microservice framework following best cloud practices with a focus on productivity.

patron CI codecov Go Report Card GoDoc GitHub release

Patron is a framework for creating microservices, originally created by Sotiris Mantzaris (https://github.com/mantzas). This fork is maintained by Beat Engineering (https://thebeat.co)

Patron is french for template or pattern, but it means also boss which we found out later (no pun intended).

The entry point of the framework is the Service. The Service uses Components to handle the processing of sync and async requests. The Service starts by default an HTTP Component which hosts the debug, alive, ready and metric endpoints. Any other endpoints will be added to the default HTTP Component as Routes. Alongside Routes one can specify middleware functions to be applied ordered to all routes as MiddlewareFunc. The service set's up by default logging with zerolog, tracing and metrics with jaeger and prometheus.

Patron provides abstractions for the following functionality of the framework:

  • service, which orchestrates everything
  • components and processors, which provide an abstraction of adding processing functionality to the service
    • asynchronous message processing (RabbitMQ, Kafka, AWS SQS)
    • synchronous processing (HTTP)
    • gRPC support
  • metrics and tracing
  • logging

Patron provides the same defaults for making the usage as simple as possible. Patron needs Go 1.14 as a minimum.

Table of Contents

Comments
  • [WIP] Add integration tests for async/amqp, trace/amqp

    [WIP] Add integration tests for async/amqp, trace/amqp

    Which problem is this PR solving?

    This PR closes #171 Add integration tests for async/amqp.

    Short description of the changes

    • Modify Github Actions definition so that it starts the rabbitmq container and network.
    • Add integration tests for the async/amqp package.
    • Add integration tests for the trace/amqp package.

    Notes

    • I'm using different queues for each package, as I came across messages from one package 'leaking' into another package's tests.

    • I will have to run some commits so I can live-debug Github Actions, hope it's okay.

    As we mentioned, we need to avoid flakiness. The tests seem to work without flakiness both locally and on Github Actions.

    $ for i in {1..50}; do
    go test -timeout 30s github.com/beatlabs/patron/async/amqp -run "^(TestConsumeAndDeliver)$" -count=1
    done | grep "^ok" | wc -l
        50 
    
    $ for i in {1..20}; do go test ./component/async/amqp -tags=integration -count=1; done | grep ^ok | wc -l
          20
    $ for i in {1..20}; do go test ./client/amqp -tags=integration -count=1; done | grep ^ok | wc -l
          20
    
    
  • Kafka: add support for active brokers in Producers

    Kafka: add support for active brokers in Producers

    Is your feature request related to a problem? Please describe

    It is necessary to check the healthiness of Kafka consumers/producers; in https://github.com/Shopify/sarama/issues/1341 it is mentioned that this could be done by checking that there are active brokers (they are checked periodically by the client anyways).

    Describe the solution

    The minimally invasive solution is to use https://godoc.org/github.com/Shopify/sarama#NewAsyncProducerFromClient in Patron and then expose the Brokers() method.

    ~This solution should work for both consumers and producers if we expose functionality of the Client.~ We will focus on the producers only.

    Additional context

    This is related to #148, in the sense that more functionality of the Kafka client library is necessary.

  • It's not possible to change deflate compression level

    It's not possible to change deflate compression level

    Is your feature request related to a problem? Please describe

    I want to have the possibility to override a default deflate level. It's hardcoded to 8 (deflateLevel = 8) in HTTP component builder. I want to be able to choose this value as it impacts the performance of my service and the size of a payload.

    Is this a bug? Please provide steps to reproduce, a failing test etc.

    It's not a bug

    Describe the solution

    I want to expose it using the env var in service.go::createHTTPComponent(), the same way as port and timeouts are exposed. The builder method is already there. b.WithDeflateLevel()

    Additional context

    I believe that it's just a good practice for a framework to have as little opinionated hardcoded values as possible. Everything or almost everything should be configurable in a good framework if we want other people to use it.

  • Proposal: Add

    Proposal: Add "path" tag for decoding URN parameters into struct fields.

    Hi there,

    Is your feature request related to a problem? Please describe

    Let’s assume we have an endpoint: UPDATE /v1/user/:id with body {"name": string, "age": int}

    Usually there is a struct that represents this request that looks like

    type UpdateUserRequest struct {
    	ID   int
    	Name string `json:"name"`
    	Age  int    `json:"age"`
    }
    

    Handlers usually do this:

    func (r *Router) updateUser(ctx context.Context, req *http.Request) (*http.Response, error) {
            var r UpdateUserRequest
            if err := req.Decode(&r); err != nil {
                return nil, errors.New("failed to decode request")
            }
    
            // check if the param is there 
            s, ok := req.Fields["id"]
    		if !ok {
    			return nil, errors.New("id is not found")
    		}
    		id, err := strconv.Atoi(s)
    		if err != nil {
    			return nil, errors.New("id is not an integer")
    		}
    
            // assigning ID to the struct
            r.ID = id
    
            // validating
            if err := r.Validate(); err != nil {
               return nil, err
            }
    
            // finally at this point it's possible to use the structure
    }
    

    In case of a request without the body ( e.g.,GET /v1/user/:id ) using req.Fields["id"] is more or less bearable.

    But in the case when a request has the body in JSON like in the example - it’s getting inconvenient ( another case when endpoint looks like POST /v1/user/:group_id/:id ).

    The actual implementation depends from case to case but what’s in common is a pretty boilerplate part of converting string representation into integers + merging with “request” struct

    Describe the solution

    I’d like to propose is using tags. Imagine that we have this struct

    type UpdateUserRequest struct {
    	ID   int    `path:"id"` <<<<-- here's a hint for Patron to map `:id` from URN onto this field
    	Name string `json:"name"`
    	Age  int    `json:"age"`
    }
    

    In this case a handler could look like

    func (r *Router) updateUser(ctx context.Context, req *http.Request) (*http.Response, error) {
            var r UpdateUserRequest
            if err := req.Decode(&r); err != nil {
                return nil, errors.New("failed to decode request")
            }
    
            // validating
            if err := r.Validate(); err != nil {
               return nil, err
            }
    
            // finally at this point it's possible to use structure
    }
    

    What do you think about this proposal?

    Thank you.

  • #277 - Fix port conflict for sixth and seventh examples

    #277 - Fix port conflict for sixth and seventh examples

    Signed-off-by: santosh [email protected]

    Which problem is this PR solving?

    Resolves #277

    Short description of the changes

    Change port for seventh example to 50007 to avoid conflict with sixth example. Also change the port in DoTimingRequest function of 1st example which uses service exposed by 7th example.

  • feat: Kafka Consumer Package

    feat: Kafka Consumer Package

    Which problem is this PR solving?

    Closes #260

    Short description of the changes

    We've had a few discussions about the best way to move forward with the task; the key outcomes were

    • Kafka's streaming logic is different than the message broker logic of AMQP and SNS/SQS. Trying to contain both under a single abstraction is not ideal.
    • Instead of maintaining the current abstraction, we've decided to introduce a new Kafka package with support for batching and
    • This new package is more transparent in regards to Sarama; exposing more its internals, plus with QoL helpers for the end-users.

    Checklist

    • [x] Component
    • [x] Retries
    • [x] Batching
    • [x] Raw Sarama Message
    • [x] Raw Sarama Config
    • [x] Update Sarama
    • [x] Failure strategy
    • [x] Metrics
    • [x] Tracing
    • [x] Unit tests
    • [x] Integration tests
  • Prometheus Exemplars Support

    Prometheus Exemplars Support

    Which problem is this PR solving?

    Enabling Prometheus Exemplars. This PR is a copy of the POC made by @ioannissavvaidis.

    Short description of the changes

    • Added support for Prometheus Exemplars

    • Updated Observarbility.md documentation

    • Enabled Prometheus Exemplars for request latency and counter metric in

      • ../client/http/http.go
      • ../client/amqp/v2/amqp.go
      • ../client/es/elasticsearch.go
      • ../client/grpc/grpc.go
      • ../client/redis/redis.go
      • ../client/sns/v2/publisher.go
      • ../client/sql/publisher.go
      • ../client/sns/v2/publisher.go
      • ../component/amqp
      • ../component/grpc
      • ../component/kafka
      • ..component/http/middleware
      • ../component/sqs
      • ../client/kafka/v2
    • Fixed a way how we record "success" metric. Mark metric entry as "success" if err == nil before we marked it as success if err != nil

  • Documentation, Comments and renaming some variables and functions (fixes #385)

    Documentation, Comments and renaming some variables and functions (fixes #385)

    Signed-off-by: Rodrigo Broggi [email protected]

    Which problem is this PR solving?

    This PR aims on enriching the whole experience of using the (amazing) examples folder. During it's development some small bugs were identified and therefore I tried to solve them as well within the scope of the PR.

    You can find more detailed comments across the changed files.

    1. Documentation enhancement with creation of diagrams and new notes in the Examples.md file Issue 385.
    2. Creation of a wrapper script called services.sh in the examples folder that helps with startup/stop of the services in the examples folder
    3. Attempt to correct two bugs that were affecting the correct functioning of the examples integration suite:
      1. panic in amqp/main.go ( bug in the actual prd code. issue 387)
      2. error in decoding in sqs/main.go#117

    Short description of the changes

    1. Documentation enhancements
    2. Minor nil pointer defensive programming
    3. update of the examples integration suite that was not working
  • Add metrics to zerolog and std logger

    Add metrics to zerolog and std logger

    Which problem is this PR solving?

    Resolves #513

    Short description of the changes

    Add a prometheus counter to the zerolog and std log package functions and unit test to see if the counter is incremented.

  • Feature: new HTTP server metrics with full status code

    Feature: new HTTP server metrics with full status code

    Which problem is this PR solving?

    Mainly:

    • Prometheus metrics to have the full HTTP status code

    We currently use Jaeger metrics which only sport 5xx, 2xx and the like; having the full status code in metrics is necessary for tailoring alerts and making more specific Grafana panels.

    This PR models the new metrics around the existing server-side gRPC metrics.

    Secondarily:

    • allowing to disable Jaeger/OpenTracing metrics (NOTE: this also means that the failures at parsing JAEGER_* environment variables now would happen when Run() is called instead of the constructor)
    • do not cache only the last Write in the response writer middleware, but all writes
    • do not cache written response if unnecessary for error logging purposes

    Short description of the changes

    We currently have:

    1. {microservice_name}_http_requests_total, {microservice_name}_http_requests_latency - these are provided by the Jaeger package and have labels endpoint="GET-/hello/world" and status_code="2xx"
    2. component_grpc_handled_total and component_grpc_handled_seconds with labels grpc_code="OK", grpc_method="CreateMyEvent", grpc_service="myservice.Service", grpc_type="unary"

    With this PR we would have:

    1. same as (1) before, but can be disabled by using WithoutJaeger() option
    2. same as (2) before
    3. component_http_handled_total and component_http_handled_seconds with labels status_code="200", method="GET", path="/hello/world", which follows the style of (2)
  • Errors with structured logging fields

    Errors with structured logging fields

    Is your feature request related to a problem? Please describe

    In the case where errors are returned to patron and patron logs them it would be really useful if there was a way to pass fields for the purpose of structured logging.

    Is this a bug? Please provide steps to reproduce, a failing test etc.

    Not a bug.

    Describe the solution

    An errors.WithFields interface could be used to see if there are any additional fields available to log.

    Additional context

    I've got a POC PR to illustrate the concept, the objective of the PR is to give context for discussions whether this is a good idea or not.

  • Bump github.com/stretchr/testify from 1.8.0 to 1.8.1

    Bump github.com/stretchr/testify from 1.8.0 to 1.8.1

    Bumps github.com/stretchr/testify from 1.8.0 to 1.8.1.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
  • Use docker-compose healthchecks

    Use docker-compose healthchecks

    Is your feature request related to a problem? Please describe

    In order to have our docker-compose files return only when all things have started, it would be nice to introduce healthchecks. This way starting up the environment will return only if all containers have started succesfully.

    Describe the solution

    Introduce healthchecks in our docker-compose files and remove any sleeps that have been introduced.

  • Patron service set up is broken

    Patron service set up is broken

    Is your feature request related to a problem? Please describe

    With the removal of the builder pattern, the setup of Patron has become a one-liner with the New() function. Previously though, by using the builder, we created a builder first which set up everything and then we created the service and ran it.

    The result is that logging and tracing are not set up correctly and the whole main function in the service is using the default logger.

    Is this a bug? Please provide steps to reproduce, a failing test etc.

    Just by creating a service, the problem is apparent.

    Describe the solution

    We need to re-introduce the two-step approach of setting things up first and then starting the service. We might even think about introducing the builder again.

    Another option, in order to avoid a builder pattern, is to keep the New() function which sets up a service and provides the components to run in the Run() method the functional options. This would give us the flexibility of the functional option pattern without the extra builder struct.

  • Add Deterministic ReadyCheck implementation

    Add Deterministic ReadyCheck implementation

    Is your feature request related to a problem? Please describe

    In kafka component we have a feature which provides info when a consumer is reached current offset (it closes a channel)

    simple.WithDurationOffset(since time.Duration)
    simple.WithNotificationOnceReachingLatestOffset(ch chan<- struct{})
    

    This is usually used in a readiness check to wait until consumer is ready and then a service is ready to process incoming requests. Readiness check is a simple function interface

    // ReadyCheckFunc defines a function type for implementing a readiness check.
    type ReadyCheckFunc func() ReadyStatus
    

    Currently, every time when one wants to use this feature they have to implement boilerplate readiness check.

    Describe the solution

    1. Add new package patron/probe independent from HTTP. It can be used then in gRPC probes.
    2. Extract probes related code into this new package in patron/probe/probe.go
    3. Add deterministic readiness check implementation in this new package in patron/probe/deterministic.go.

    Additional context

    New package with 2 files

    patron/probe
    -- probe.go
    -- deterministic.go
    

    Move probe related code into probe/probe.go file from component/http/v2/check.go. Leave code related to http routes there.

    package probe
    
    // AliveStatus type representing the liveness of the service.
    type AliveStatus int
    
    // ReadyStatus type.
    type ReadyStatus int
    
    const (
    	// Alive represents a state defining an Alive state.
    	Alive AliveStatus = 1
    	// Unhealthy represents an unhealthy alive state.
    	Unhealthy AliveStatus = 2
    
    	// Ready represents a state defining a Ready state.
    	Ready ReadyStatus = 1
    	// NotReady represents a state defining a NotReady state.
    	NotReady ReadyStatus = 2
    
    	// AlivePath of the component.
    	AlivePath = "/alive"
    	// ReadyPath of the component.
    	ReadyPath = "/ready"
    )
    
    // ReadyCheckFunc defines a function type for implementing a readiness check.
    type ReadyCheckFunc func() ReadyStatus
    
    // LivenessCheckFunc defines a function type for implementing a liveness check.
    type LivenessCheckFunc func() AliveStatus
    

    Add another file probe/deterministic.go with the following deterministic ready check implementation.

    package probe
    
    import (
    	"sync"
    	"time"
    
    	"github.com/beatlabs/patron/log"
    )
    
    // DeterministicReadyCheck waits until all the defined channels are closed
    type DeterministicReadyCheck struct {
    	sync.Mutex
    	checks map[string]*readyCheck
    	status v2.ReadyStatus
    }
    
    // NewDeterministicReadyCheck creates new deterministic check
    func NewDeterministicReadyCheck() *DeterministicReadyCheck {
    	return &DeterministicReadyCheck{
    		checks: make(map[string]*readyCheck),
    		status: v2.NotReady,
    	}
    }
    
    // AddCheck add new channel to watch when it's closed
    func (r *DeterministicReadyCheck) AddCheck(name string, ch chan struct{}) *DeterministicReadyCheck {
    	r.checks[name] = &readyCheck{name: name, ch: ch, startedAt: time.Now()}
    	return r
    }
    
    // Status ask when all the checks are ready
    func (r *DeterministicReadyCheck) Status() v2.ReadyStatus {
    	r.Lock()
    	defer r.Unlock()
    
    	if r.status == v2.NotReady {
    		for _, chk := range r.checks {
    			if !chk.isReady() {
    				return r.status
    			}
    		}
    		r.status = v2.Ready
    	}
    
    	return r.status
    }
    
    type readyCheck struct {
    	name      string
    	ch        <-chan struct{}
    	startedAt time.Time
    	closed    bool
    }
    
    func (r *readyCheck) isReady() bool {
    	if r.closed {
    		return true
    	}
    	// if channel was closed, set the flag and log how long it took
    	select {
    	case _, ok := <-r.ch:
    		if !ok {
    			r.closed = true
    			log.Infof("readiness %s: %v", r.name, time.Since(r.startedAt))
    			return true
    		}
    	default:
    	}
    	return false
    }
    
Goal is a toolkit for high productivity web development in Go language in the spirit of Revel Framework that is built around the concept of code generation.

Goal Goal is a set of tools for high productivity web development in Go language. Goal, being mostly inspired by Revel Framework and its discussions,

Sep 27, 2021
go-zero is a web and rpc framework written in Go. It's born to ensure the stability of the busy sites with resilient design. Builtin goctl greatly improves the development productivity.
go-zero is a web and rpc framework written in Go. It's born to ensure the stability of the busy sites with resilient design. Builtin goctl greatly improves the development productivity.

go-zero English | 简体中文 0. what is go-zero go-zero is a web and rpc framework that with lots of engineering practices builtin. It’s born to ensure the

Jan 2, 2023
go-zero is a web and rpc framework that with lots of engineering practices builtin.
go-zero is a web and rpc framework that with lots of engineering practices builtin.

go-zero is a web and rpc framework that with lots of engineering practices builtin. It’s born to ensure the stability of the busy services with resilience design, and has been serving sites with tens of millions users for years.

Jan 6, 2023
Gearbox :gear: is a web framework written in Go with a focus on high performance
Gearbox :gear: is a web framework written in Go with a focus on high performance

gearbox ⚙️ is a web framework for building micro services written in Go with a focus on high performance. It's built on fasthttp which is up to 10x fa

Jan 3, 2023
Gearbox :gear: is a web framework written in Go with a focus on high performance
Gearbox :gear: is a web framework written in Go with a focus on high performance

gearbox ⚙️ is a web framework for building micro services written in Go with a focus on high performance. It's built on fasthttp which is up to 10x fa

Dec 29, 2022
Kubewrap - kubewrap is an kubernetes command line utility with a focus to explore the kubernetes REST API's leveraging the go libraries available along with golang and cobra packages.

Kubewrap kubewrap is an kubernetes cli wrapper with a focus to explore the kubernetes REST API's leveraging the go libraries available along with gola

Nov 20, 2022
This library provides a simple framework of microservice, which includes a configurator, a logger, metrics, and of course the handler

Microservice The framework for the creation of microservices, written in Golang. (note: http microservice) Architecture microservice includes: handle

Dec 30, 2022
Best simple, lightweight, powerful and really fast Api with Golang (Fiber, REL, Dbmate) PostgreSqL Database and Clean Architecture

GOLANG FIBER API (CLEAN ARCHITECTURE) Best simple, lightweight, powerful and really fast Api with Golang (Fiber, REL, Dbmate) PostgreSqLDatabase using

Sep 2, 2022
A Microservice Toolkit from The New York Times
A Microservice Toolkit from The New York Times

Gizmo Microservice Toolkit This toolkit provides packages to put together server and pubsub daemons with the following features: Standardized configur

Dec 27, 2022
Flamingo Framework and Core Library. Flamingo is a go based framework for pluggable web projects. It is used to build scalable and maintainable (web)applications.
Flamingo Framework and Core Library. Flamingo is a go based framework for pluggable web projects. It is used to build scalable and maintainable (web)applications.

Flamingo Framework Flamingo is a web framework based on Go. It is designed to build pluggable and maintainable web projects. It is production ready, f

Jan 5, 2023
Golanger Web Framework is a lightweight framework for writing web applications in Go.

/* Copyright 2013 Golanger.com. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except

Nov 14, 2022
The jin is a simplified version of the gin web framework that can help you quickly understand the core principles of a web framework.

jin About The jin is a simplified version of the gin web framework that can help you quickly understand the core principles of a web framework. If thi

Jul 14, 2022
laravel for golang,goal,fullstack framework,api framework
laravel for golang,goal,fullstack framework,api framework

laravel for golang,goal,fullstack framework,api framework

Feb 24, 2022
Gin is a HTTP web framework written in Go (Golang).
Gin is a HTTP web framework written in Go (Golang).

Gin is a HTTP web framework written in Go (Golang). It features a Martini-like API with much better performance -- up to 40 times faster. If you need smashing performance, get yourself some Gin.

Jan 3, 2023
An ideally refined web framework for Go.

Air An ideally refined web framework for Go. High-performance? Fastest? Almost all web frameworks are using these words to tell people that they are t

Dec 15, 2022
An opinionated productive web framework that helps scaling business easier.
An opinionated productive web framework that helps scaling business easier.

appy An opinionated productive web framework that helps scaling business easier, i.e. focus on monolith first, only move to microservices with GRPC la

Nov 4, 2022
BANjO is a simple web framework written in Go (golang)

BANjO banjo it's a simple web framework for building simple web applications Install $ go get github.com/nsheremet/banjo Example Usage Simple Web App

Sep 27, 2022
beego is an open-source, high-performance web framework for the Go programming language.
beego is an open-source, high-performance web framework for the Go programming language.

Beego Beego is used for rapid development of enterprise application in Go, including RESTful APIs, web apps and backend services. It is inspired by To

Jan 1, 2023
High performance, minimalist Go web framework
High performance, minimalist Go web framework

Supported Go versions As of version 4.0.0, Echo is available as a Go module. Therefore a Go version capable of understanding /vN suffixed imports is r

Jan 2, 2023