RadonDB is an open source, cloud-native MySQL database for building global, scalable cloud services

Build Status Go Report Card codecov.io

OverView

RadonDB is an open source, Cloud-native MySQL database for unlimited scalability and performance.

What is RadonDB?

RadonDB is a cloud-native database based on MySQL, and architected in fully distributed cluster that enable unlimited scalability (scale-out), capacity and performance. It supported distributed transaction that ensure high data consistency, and leveraged MySQL as storage engine for trusted data reliability. RadonDB is compatible with MySQL protocol, and sup-porting automatic table sharding as well as batch of automation feature for simplifying the maintenance and operation workflow.

Features

  • Automatic Sharding
  • Auditing and Logging
  • Parallel Execution: Parallel Query, Parallel DML and Parallel DDL
  • Parallel CHECKSUM TABLE: Gives same results as MySQL
  • Distributed Transaction: Snapshot Isolation
  • Distributed Joins: Sort-Merge Join, Nested-Loop Join
  • Distributed Full Text Search
  • Multi Tenant by Database
  • Prepared SQL Statement
  • JSON

Documentation

For guidance on installation, deployment, and administration, see our Documentation.

Architecture

Overview

RadonDB is a new generation of distributed relational database (MyNewSQL) based on MySQL. It was designed to create the open-source database our developers would want to use: one that has features such as financial high availability、 large-capacity database、automatic plane split table、 scalable and strong consistency, this guide sets out to detail the inner-workings of the radon process as a means of explanation.

SQL Layer

SQL support

On SQL syntax level, RadonDB Fully compatible with MySQL.You can view all of the SQL features RadonDB supports here radon_sql_statements_manual

SQL parser, planner, executor

After your SQL node receives a SQL request from a mysql client via proxy, RadonDB parses the statement, creates a query plan, and then executes the plan.

                                                                +---------------+
                                                    x---------->|node1_Executor |
                            +--------------------+  x           +---------------+
                            |      SQL Node      |  x
                            |--------------------|  x
+-------------+             |     sqlparser      |  x           +---------------+
|    query    |+----------->|                    |--x---------->|node2_Executor |
+-------------+             |  Distributed Plan  |  x           +---------------+
                            |                    |  x
                            +--------------------+  x
                                                    x           +---------------+
                                                    x---------->|node3_Executor |
                                                                +---------------+

Parsing

Received queries are parsed by sqlparser (which describes the supported syntax by mysql) and generated Abstract Syntax Trees (AST).

Planning

With the AST, RadonDB begins planning the query's execution by generating a tree of planNodes. This step also includes steps analyzing the client's SQL statements against the expected AST expressions, which include things like type checking.

You can see the a query plan generates using EXPLAIN(At this stage we only use EXPLAIN to analysis Table distribution).

Excuting

Executing an Executor in a storage layer in Parallel with a Distributed Execution Plan.

SQL with Transaction

The SQL node is stateless, but in order to guarantee transaction Snapshot Isolation, it is currently a write-multiple-read mode.

Transaction Layer

Distributed transaction

RadonDB provides distributed transaction capabilities. If the distributed executor at different storage nodes and one of the nodes failed to execute, then operation of the rest nodes will be rolled back, This guarantees the atomicity of operating across nodes and makes the database in a consistent state.

Isolation Levels

RadonDB achieves the level of SI (Snapshot Isolation) at the level of consistency. As long as a distributed transaction has not commit, or if some of the partitions have committed, the operation is invisible to other transactions.

Transaction with SQL Layer

The SQL node is stateless, but in order to guarantee transaction Snapshot Isolation, it is currently a write-multiple-read mode.

Issues

The integrated github issue tracker is used for this project.

License

RadonDB is released under the GPLv3. See LICENSE

Owner
RadonDB
RadonDB = NewSQL + MySQL
RadonDB
Comments
  • radon如何支持外部访问?

    radon如何支持外部访问?

    1. 参考文档,rest-api的确可以实现给后端MySQL添加本地用户,但radon只能在本机访问 报错如下: 2018/05/15 17:17:45.892306 auth.go:139: [ERROR] proxy: auth.user[root].failed(password.invalid):want[[89 223 240 70 44 7 2 202 154 74 149 248 53 42 238 70 192 194 109 206]]!=got[[77 34 129 174 3 224 34 45 152 251 202 58 202 169 175 254 225 178 11 148]] 2018/05/15 17:17:45.892361 server.go:153: [WARNING] server.user[root].auth.check.failed
  • planner: fix column name with no quote when scatter to backends #489

    planner: fix column name with no quote when scatter to backends #489

    [summary] When scatter sql to backends, the column name should be with quote '' e.g.: create table db.t(keyint primary key,colint); For the ddl with rename table, we remove all quotes \'' to uniform standars. e.g.: alter table t1 rename t2.

    [test case] src/planner/ddl_plan_test.go

    [patch codecov] src/planner/ddl_plan.go 95.1% coverage: 89.8% of statements

  • create table with COLLATE error

    create table with COLLATE error

    when create table with COLLATE,it is wrong: mysql> CREATE TABLE data_gps_htest ( -> CAR_ID VARCHAR(32) NULL DEFAULT NULL COLLATE 'utf8_bin', -> IO_STTS VARCHAR(16) NULL DEFAULT NULL COLLATE 'utf8_bin', -> CAR_STTS VARCHAR(32) NULL DEFAULT NULL COLLATE 'utf8_bin', -> ALARM_STTS VARCHAR(32) NULL DEFAULT NULL COLLATE 'utf8_bin', -> EXTS VARCHAR(4000) NULL DEFAULT NULL COLLATE 'utf8_bin', -> GPS_TYPE VARCHAR(18) NULL DEFAULT NULL COLLATE 'utf8_bin', -> SGN_STTS VARCHAR(32) NULL DEFAULT NULL COLLATE 'utf8_bin', -> TMN_MILE INT(11) NULL DEFAULT NULL, -> INDEX DATA_GPS_H_IDX (CAR_ID) -> ) -> COLLATE='utf8_bin' -> ROW_FORMAT=COMPRESSED; ERROR 1149 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use, syntax error at position 79 near 'collate'

  • Remote access reported

    Remote access reported "error 2013: lost connection to mysql server"

    As mentioned in title, remote access failed, and radon server would log error server.unpack.auth.error: invalid authPluginName, got caching_sha2_password but only support mysql_native_password.

    I guess it may related to the version of maraidb latest, any solution?

    Besides, in local machine, enter into mysql command line, execute use mysql, would report ERROR 1227 (42000): Access denied; lacking privileges for database mysql

  • [improve] when execute the JOIN which visit multiple backends, client sometimes hangs

    [improve] when execute the JOIN which visit multiple backends, client sometimes hangs

    When the twopc-enable is true in the config,and execute the join which visit multiple backends,the client will hang, as the problem appears in TPCC. To avoid doing a lot of preparation about the TPCC environment,a simple example is as follows:

    create database join_hang;
    use join_hang;
    CREATE TABLE `a` (`i` int(11) DEFAULT NULL,`j` varchar(255) DEFAULT NULL ) PARTITION BY HASH(i);
    CREATE TABLE `b` (`i` int(11) DEFAULT NULL,`j` varchar(255) DEFAULT NULL ) PARTITION BY HASH(i);
    insert into a(i,j) values(1,'a'),(2,'a'),(3,'a'),(4,'a');
    insert into b(i,j) values(1,'a'),(2,'a'),(3,'a'),(4,'a');
    
    mysql> select * from a;
    +------+------+
    | i    | j    |
    +------+------+
    |    1 | a    |
    |    4 | a    |
    |    3 | a    |
    |    2 | a    |
    +------+------+
    4 rows in set (0.01 sec)
    
    mysql> select * from b;
    +------+------+
    | i    | j    |
    +------+------+
    |    1 | a    |
    |    4 | a    |
    |    3 | a    |
    |    2 | a    |
    +------+------+
    4 rows in set (0.01 sec)
    
    mysql> select a.i, a.j, b.i, b.j from a join b where a.i<2 and b.j='a';
    +------+------+------+------+
    | i    | j    | i    | j    |
    +------+------+------+------+
    |    1 | a    |    1 | a    |
    |    1 | a    |    4 | a    |
    |    1 | a    |    3 | a    |
    |    1 | a    |    2 | a    |
    +------+------+------+------+
    4 rows in set (0.01 sec)
    
    mysql> select a.i, a.j, b.i, b.j from a join b where a.i<2 and b.j='a';
    
    It will always hang for a long time.
    
  • [feature] GLOBAL and SINGLE table syntax

    [feature] GLOBAL and SINGLE table syntax

    Summary

    We have 3 type tables:

    1. global table: every backend has
    CREATE TABLE t1(a int...) GLOBAL;
    
    1. single table: on one deterministic backend
    CREATE TABLE t1(a int...) SINGLE;
    
    1. partition table: by hash
    CREATE TABLE t1(a int...) PARTITION BY HASH(a);
    
    1. If table without GLOBAL/SINGLE and PARITION BY HASH, we will do partition table conversion by PRIMARY KEY:
    CREATE TABLE t1(a int parimary key) ;
    

    to

    CREATE TABLE t1(a int parimary key) PARTITION BY HASH(a);
    

    WIP

    • [x] Table type supports (@BohuTANG )
    • [ ] Single Router (@zhyass )
    • [ ] Single Table Planner (@zhyass )
    • [ ] Single Table Rebalance (@zhyass )
    • [ ] Document (@zhyass )
    • [ ] Show Create Table with Table Type (@andyli029 )
  • *: enhance set statement syntax #629

    *: enhance set statement syntax #629

    [summary]

    1. reference: https://dev.mysql.com/doc/refman/5.7/en/set-variable.html
    SET variable = expr [, variable = expr] ...
    
    variable: {
        user_var_name
      | param_name
      | local_var_name
      | {GLOBAL | @@GLOBAL.} system_var_name
      | [SESSION | @@SESSION. | @@] system_var_name
    }
    
    1. reference: https://dev.mysql.com/doc/refman/5.7/en/set-transaction.html
    SET [GLOBAL | SESSION] TRANSACTION
        transaction_characteristic [, transaction_characteristic] ...
    
    transaction_characteristic: {
        ISOLATION LEVEL level
      | access_mode
    }
    
    level: {
         REPEATABLE READ
       | READ COMMITTED
       | READ UNCOMMITTED
       | SERIALIZABLE
    }
    
    access_mode: {
         READ WRITE
       | READ ONLY
    }
    

    [test case] src/proxy/set_test.go src/vendor/github.com/xelabs/go-mysqlstack/sqlparser/set_test.go [patch codecov] src/proxy/set.go 94.6% src/vendor/github.com/xelabs/go-mysqlstack/sqlparser/ast.go 88.0%

  • Unknown command '\

    Unknown command '\"' & '\'' can't support

    table struct: CREATE TABLE wp_options ( option_id bigint(20) unsigned NOT NULL AUTO_INCREMENT, option_name varchar(191) DEFAULT NULL, option_value longtext NOT NULL, autoload varchar(20) NOT NULL DEFAULT 'yes', PRIMARY KEY (option_id), UNIQUE KEY option_name (option_name) ) single;

    SQL: INSERT INTO wp_options(option_id,option_name,option_value,autoload) VALUES(38,"active_plugins","a:4:{i:0;s:23:"anti-spap";}","yes");

    Radon err:

    2019/09/19 19:58:14.550460 query.go:69: [ERROR] query[INSERT INTO wp_options(option_id,option_name,option_value,autoload) VALUES(38,"active_plugins","a:4:{i:0;s:23:"anti-spap"].parser.error: syntax error at position 135 near 'a:4:{i:0;s:23:"anti-spap"' 2019/09/19 19:58:14.550505 server.go:240: [ERROR] server.handle.query.from.session[75].error:You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use, syntax error at position 135 near 'a:4:{i:0;s:23:"anti-spap"' (errno 1149) (sqlstate 42000).query[INSERT INTO wp_options(option_id,option_name,option_value,autoload) VALUES(38,"active_plugins","a:4:{i:0;s:23:"anti-spap"]

  • ctl: add more details for privilege options #470

    ctl: add more details for privilege options #470

    [summary] Now when we create user, it has all privileges, so we should give host and [select|insert|update|delete] options to limit the privilege.

    [test case] TestCtlV1CreateUserError TestCtlV1CreateUserHostAndPriv TestCtlV1AlterUser TestCtlV1DropUser

    [patch codecov] ctl/v1/user.go (96.6%)

  • proxy: add reshard handle #436

    proxy: add reshard handle #436

    [summary]

    Syntax: RADON RESHARD tbl_name TO new_tbl_name

    You can use RADON RESHARD to shift data from one SINGLE table to another PARTITION table, the cmd execute the shift cmd and will return immediately, the shift will run in background on other goroutine.

    NOTICE: the SINGLE table with the primary key can be partitioned.

    TODO: the shift progress bar will call other interface

    [test case]

    src/proxy/admin_reshard_test.go src/proxy/admin_shift_test.go src/router/frm_test.go

    [patch codecov] src/proxy/admin_reshard.go: 98.4% src/proxy/admin_shift.go: 81.8% src/router/frm.go: 100%

  • *: use errgroup to handle error under concurrency #480

    *: use errgroup to handle error under concurrency #480

    https://github.com/radondb/radon/issues/480#issuecomment-636710899

    [summary] Package errgroup provides synchronization, error propagation, and Context cancelation for groups of goroutines working on subtasks of a common task. We can use the errgroup to handle error under concurrency. [test case] N/A [patch codecov] src/backend/txn.go 89.8% src/backend/xa.go 91.1% src/executor/engine/join_engine.go 95.7% src/executor/engine/union_engine.go 93.9%

  • default username and passwd?

    default username and passwd?

    I create a sigle node radon with mysql5.7 follw docs how_to_build_and_run_radon.md

    when Step5. Connect mysql client to radon respone " Access denied for user 'root' "

    It means my username or passwd incorrect? what is the default username and passwd of radondb

Beerus-DB: a database operation framework, currently only supports Mysql, Use [go-sql-driver/mysql] to do database connection and basic operations

Beerus-DB · Beerus-DB is a database operation framework, currently only supports Mysql, Use [go-sql-driver/mysql] to do database connection and basic

Oct 29, 2022
TiDB is an open source distributed HTAP database compatible with the MySQL protocol
TiDB is an open source distributed HTAP database compatible with the MySQL protocol

Slack Channel Twitter: @PingCAP Reddit Mailing list: lists.tidb.io For support, please contact PingCAP What is TiDB? TiDB ("Ti" stands for Titanium) i

Jan 9, 2023
An open-source graph database
An open-source graph database

Cayley is an open-source database for Linked Data. It is inspired by the graph database behind Google's Knowledge Graph (formerly Freebase). Documenta

Dec 31, 2022
LinDB is an open-source Time Series Database which provides high performance, high availability and horizontal scalability.
LinDB is an open-source Time Series Database which provides high performance, high availability and horizontal scalability.

LinDB is an open-source Time Series Database which provides high performance, high availability and horizontal scalability. LinDB stores all monitoring data of ELEME Inc, there is 88TB incremental writes per day and 2.7PB total raw data.

Jan 1, 2023
Virsas-mod-db - Quick way to init mysql, postgres and redis connection from multiple services without duplicating the code

Quick way to init mysql, postgres and redis connection from multiple services without duplicating the code.

Jan 23, 2022
A MySQL-compatible relational database with a storage agnostic query engine. Implemented in pure Go.

go-mysql-server is a SQL engine which parses standard SQL (based on MySQL syntax) and executes queries on data sources of your choice. A simple in-memory database and table implementation are provided, and you can query any data source you want by implementing a few interfaces.

Dec 27, 2022
Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding.

Vitess is a database clustering system for horizontal scaling of MySQL through generalized sharding.

Jan 4, 2023
Run MySQL Database on Docker

Run MySQL Database on Docker cd <path>/resources/docker sudo docker-compose up (sudo for linux) This will start a container MySQL Database running on

Jan 1, 2022
Native GraphQL Database with graph backend
Native GraphQL Database with graph backend

The Only Native GraphQL Database With A Graph Backend. Dgraph is a horizontally scalable and distributed GraphQL database with a graph backend. It pro

Jan 4, 2023
Set out to become the de facto open-source alternative to MongoDB

MangoDB MangoDB is set out to become the de facto open-source alternative to MongoDB. MangoDB is an open-source proxy, which converts MongoDB wire pro

Dec 29, 2022
Owl is a db manager platform,committed to standardizing the data, index in the database and operations to the database, to avoid risks and failures.

Owl is a db manager platform,committed to standardizing the data, index in the database and operations to the database, to avoid risks and failures. capabilities which owl provides include Process approval、sql Audit、sql execute and execute as crontab、data backup and recover .

Nov 9, 2022
This is a simple graph database in SQLite, inspired by "SQLite as a document database".

About This is a simple graph database in SQLite, inspired by "SQLite as a document database". Structure The schema consists of just two structures: No

Jan 3, 2023
Hard Disk Database based on a former database

Hard Disk Database based on a former database

Nov 1, 2021
Simple key value database that use json files to store the database

KValDB Simple key value database that use json files to store the database, the key and the respective value. This simple database have two gRPC metho

Nov 13, 2021
Scalable datastore for metrics, events, and real-time analytics

InfluxDB InfluxDB is an open source time series platform. This includes APIs for storing and querying data, processing it in the background for ETL or

Jan 5, 2023
Scalable datastore for metrics, events, and real-time analytics

InfluxDB InfluxDB is an open source time series platform. This includes APIs for storing and querying data, processing it in the background for ETL or

Jan 4, 2023
Couchbase - distributed NoSQL cloud database

couchbase Couchbase is distributed NoSQL cloud database. create Scope CREATE SCO

Feb 16, 2022
A distributed MySQL binlog storage system built on Raft
A distributed MySQL binlog storage system built on Raft

What is kingbus? 中文 Kingbus is a distributed MySQL binlog store based on raft. Kingbus can act as a slave to the real master and as a master to the sl

Dec 31, 2022
The MySQL Cluster Autopilot Management with GTID and Raft
The MySQL Cluster Autopilot Management with GTID and Raft

Xenon Overview Xenon is a MySQL HA and Replication Management tool using Raft protocol. Xenon has many cool features, such as: Fast Failover with no l

Jan 3, 2023