A db proxy for distributed transaction, read write splitting and sharding! Support any language! It can be deployed as a sidecar in a pod.

DBPack

LICENSE codecov

dbpack.svg

DBPack means a database cluster tool pack. It can be deployed as a sidecar in a pod, it shields complex basic logic, so that business development does not need to rely on a specific SDK, simplifying the development process and improving development efficiency.

Feature

  • support MYSQL protocol
  • simple and easy distributed transaction solution inspired by kubernetes
  • support read write splitting，supports custom SQL routing via Hint
  • deployed as a sidecar，support any language
  • sharding: support query on sharded table, support order by，support limit
  • more features on the road

Metrics

using prometheus metrics, default metric server run at :18888

Documentation

Requirements

  • Go 1.17 or higher
  • MYSQL (5.7+)

Contribution Guide

Architecture

architecture

  • Listener: responsible for parsing the SQL protocol.
  • Executor: responsible for forwarding SQL requests to the backend physical DB.
  • Filter: responsible for metrics statistics, SQL interception, encryption and decryption of sensitive information, etc.
  • ConnectionFilter: process the intercepted SQL on the connection.

WeChat group

To enter the WeChat discussion group, please scan the QR code below and reply with the code "join the group".

image-20220427091848831

License

DBPack is licensed under the Apache License, Version 2.0.

Owner
CECloud Open Source
null
https://github.com/CECTC/dbpack https://cectc.github.io/dbpack-doc/
Comments
  • Test: unit test for `FormatTimeMillis`

    Test: unit test for `FormatTimeMillis`

    Added unit-tests for FormatTimeMillis func

    ref: https://github.com/cectc/dbpack/issues/70

    Ⅰ. Describe what this PR did

    Adding unit-tests for FormatTimeMillis func

    Ⅱ. Does this pull request fix one issue?

    I am not sure, but i think yes

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    tests added

    Ⅳ. Describe how to verify it

    Ⅴ. Special notes for reviews

    this PR is ok for you, i will continue for others, let me know if you have any comments for PR

  • feat: add prometheus metric

    feat: add prometheus metric

    ref: https://github.com/cectc/dbpack/issues/25

    Ⅰ. Describe what this PR did

    add prometheus metric for dbpack. opentelemetry metric currently unstable image

    Ⅱ. Does this pull request fix one issue?

    #25

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    no need tests

    Ⅳ. Describe how to verify it

    in my computer, i verify it

    Ⅴ. Special notes for reviews

    connection metric filter config image

  • test: unit test for `pkg/filter/dt/executor.go`

    test: unit test for `pkg/filter/dt/executor.go`

    Is your feature request related to a problem? Please describe. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

    Describe the solution you'd like A clear and concise description of what you want to happen.

    Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

    Additional context Add any other context or screenshots about the feature request here.

  • feat: add python examples

    feat: add python examples

    Signed-off-by: zhu733756 [email protected]

    ref: https://github.com/CECTC/dbpack/issues/31

    Ⅰ. Describe what this PR did

    add python examples

    Ⅱ. Does this pull request fix one issue?

    https://github.com/CECTC/dbpack/issues/31

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    No need

    Ⅳ. Describe how to verify it

    See README

    Ⅴ. Special notes for reviews

    None.

  • support tcc branch commit & rollback

    support tcc branch commit & rollback

    ref: https://github.com/cectc/dbpack/issues/12

    Ⅰ. Describe what this PR did

    support tcc mode distributed transaction

    Ⅱ. Does this pull request fix one issue?

    #12

  • fix: in tcc mode, json.Unmarshal requestContext.Headers failed

    fix: in tcc mode, json.Unmarshal requestContext.Headers failed

    ref: https://github.com/CECTC/dbpack/issues/12 https://github.com/cectc/dbpack/issues/194

    Ⅰ. Describe what this PR did

    1. requestContext.Headers unmarshal failed
    2. Incorrect host passed in
    3. Got wrong QueryString

    Ⅱ. Does this pull request fix one issue?

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    Ⅳ. Describe how to verify it

    verified by tcc sample image

    image

    Ⅴ. Special notes for reviews

  • add unit test cases for `convertAssignRows`

    add unit test cases for `convertAssignRows`

    ref: https://github.com/cectc/dbpack/issues/

    Ⅰ. Describe what this PR did

    Ⅱ. Does this pull request fix one issue?

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    Ⅳ. Describe how to verify it

    Ⅴ. Special notes for reviews

  • feat: add readiness and liveness probe

    feat: add readiness and liveness probe

    Signed-off-by: hzliangbin [email protected]

    ref: https://github.com/cectc/dbpack/issues/52

    Ⅰ. Describe what this PR did

    add readiness and liveness probe

    Ⅱ. Does this pull request fix one issue?

    #52

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    Ⅳ. Describe how to verify it

    Ⅴ. Special notes for reviews

  • WIP feat: intergrate opentelemetry tracing

    WIP feat: intergrate opentelemetry tracing

    ref: https://github.com/cectc/dbpack/issues/

    Ⅰ. Describe what this PR did

    #50

    • add opentelemetry support in dbpack

    Ⅱ. Does this pull request fix one issue?

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    Ⅳ. Describe how to verify it

    Ⅴ. Special notes for reviews

  • [WIP] feature: support delete on sharded table

    [WIP] feature: support delete on sharded table

    ref: https://github.com/cectc/dbpack/issues/44

    Ⅰ. Describe what this PR did

    Todo: Need to handle distributed transactions across databases

    Ⅱ. Does this pull request fix one issue?

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    there are unit test cases and integration test cases

    Ⅳ. Describe how to verify it

    check ci result

    Ⅴ. Special notes for reviews

  • Test: unit test for pkg/misc/other.go

    Test: unit test for pkg/misc/other.go

    ref: https://github.com/cectc/dbpack/issues/68

    Ⅰ. Describe what this PR did

    Added unit-test for pkg/misc/other.go

    Ⅱ. Does this pull request fix one issue?

    yes

    Ⅲ. Why don't you add test cases (unit test/integration test)?

    Ⅳ. Describe how to verify it

    Ⅴ. Special notes for reviews

  • refactor status api

    refactor status api

    Status should be displayed separately by ApplicationID

    https://github.com/CECTC/dbpack/blob/6611c17af707f8a95f67e0cad74bd197e77c084a/pkg/http/status.go#L57

  • Saga and undo logic

    Saga and undo logic

    Is there are support for sagas with compensation logic ?

    I used to work on these types of 2PC systems and the holding of locks on DB’s is imho only allowed on Databases your own . Even with a large organisation with many teams where each team controls their own database, holding locks on more than db can be disallowed.

    the alternative can be sagas where the transactions do not use locks , and is any db refuses a mutation, then the 2PC must execute compensation logic and either undo the transaction or some other form of compensation.

    Are you working on this. I lock approach or interested in pursuing it ?

    It’s a solution to when different parties need to perform business transactions across theirs and others databases.

    if you would like to discuss more in depth on this I am open to collaborating. Telegram is in my GitHub profile.

Related tags
Microservices kubernetes database microservice event-driven sharding cloud-native micro-services tcc sidecar at 2pc distributed-transaction read-write-splitting database-mesh
Example golang microservices deployed on kubernetes.
 Example golang microservices deployed on kubernetes.

Tech Stack Golang RabbitMQ Docker K8S MongoDB Services There are two services which communicate via http(sync) and rabbitmq(async). Services opened to

Sep 6, 2022
Blog microservices deployed in an Istio-enabled kubernetes cluster
 Blog microservices deployed in an Istio-enabled kubernetes cluster

blog-microservices blog microservices deployed in an Istio-enabled kubernetes cl

Dec 28, 2022
A microservice gateway developed based on golang.With a variety of plug-ins which can be expanded by itself, plug and play. what's more,it can quickly help enterprises manage API services and improve the stability and security of API services.
 A microservice gateway developed based on golang.With a variety of plug-ins which can be expanded by itself, plug and play. what's more,it can quickly help enterprises manage API services and improve the stability and security of API services.

Goku API gateway is a microservice gateway developed based on golang. It can achieve the purposes of high-performance HTTP API forwarding, multi tenant management, API access control, etc. it has a powerful custom plug-in system, which can be expanded by itself, and can quickly help enterprises manage API services and improve the stability and security of API services.

Dec 29, 2022
Global Financial Transaction Network Services

Global Financial Transaction Network Services This code was developed at IBM during 2017-2020, and contributed to open source in September 2021. Overv

Oct 9, 2022
A Micro-UTP, plug-able sanity checker for any on-prem JFrog platform instance

hello-frog About this plugin This plugin is a template and a functioning example for a basic JFrog CLI plugin. This README shows the expected structur

Dec 7, 2021
A gateway to expose s3 standard API for any storage type.

s3-gateway this project is used to be compatible with any other storage type. there are tons of object storage services in the internat. However, many

Nov 28, 2021
Omh-users-management - A go microservice that can enables us to create, modify, fetch, and delete users

Users Management System A go microservice that can enables us to create, modify, fetch, and delete users Usage To Run the application $ go run main.

Jan 20, 2022
Just a quick demo of how you can use automatically generated protobuffer and gRPC code from buf.build

buf.build demo The purpose of this repository is to demonstrate how to use the services offered by buf.build for hosting protobuffer definitions and a

Jan 4, 2022
Go-explosion - Distributed indexing and searching in Go/Golang
 Go-explosion - Distributed indexing and searching in Go/Golang

go-explosion Distributed indexing and searching in Go/Golang. This library does

Jan 1, 2022
gRPC to JSON proxy generator following the gRPC HTTP spec
 gRPC to JSON proxy generator following the gRPC HTTP spec

The gRPC-Gateway is a plugin of the Google protocol buffers compiler protoc. It reads protobuf service definitions and generates a reverse-proxy server which translates a RESTful HTTP API into gRPC. This server is generated according to the google.api.http annotations in your service definitions.

Jan 3, 2023
This project extends the go-chi router to support OpenAPI 3, bringing to you a simple interface to build a router conforming your API contract.

Go OpenAPI This project extends the go-chi router to support OpenAPI 3, bringing to you a simple interface to build a router conforming your API contr

Mar 27, 2022
Go Micro is a framework for distributed systems development

Go Micro Go Micro is a framework for distributed systems development. Overview Go Micro provides the core requirements for distributed systems develop

Jan 3, 2023
NewSQL distributed storage database based on micro service framework
 NewSQL distributed storage database based on micro service framework

QLite 是基于微服务的 NewSQL 型数据库系统，与传统的一体化数据库不同，该系统将本该内置的多种数据结构（STL）拆分成多个服务模块，每个模块都是独立的一个节点，每个节点都与其主网关进行连接，从而形成分布式存储结构。

Jun 19, 2022
CNCF Jaeger, a Distributed Tracing Platform

Jaeger - a Distributed Tracing System Jaeger, inspired by Dapper and OpenZipkin, is a distributed tracing platform created by Uber Technologies and do

Jan 2, 2023
Go Micro: a standard library for distributed systems development

Go Micro Go Micro is a standard library for distributed systems development. Ove

Oct 4, 2022
Labs for MIT 6.824 Distributed Systems (Spring 2020)

6.824-2020-labs This repo contains labs for MIT 6.824 Distributed Systems (Spring 2020) The master branch contains the source code with no implementat

Jan 18, 2022
Demonstrate a bounded context distributed over multiple repositories. In `go`

Contextive Demo - Go - Service This repository illustrates the use of Contextive in an environment where multiple repositories are part of the same bo

Feb 12, 2022
This tool generates Go language bindings of services in protobuf definition files for go-kit

protoc-gen-go-kit This tool generates Go language bindings of services in protobuf definition files for go-kit. Installation $ go install github.com/x

Nov 9, 2021
Microservice Boilerplate for Golang with gRPC and RESTful API. Multiple database and client supported
 Microservice Boilerplate for Golang with gRPC and RESTful API. Multiple database and client supported

Go Microservice Starter A boilerplate for flexible Go microservice. Table of contents Features Installation Todo List Folder Structures Features: Mult

Jul 28, 2022