A Golang implemented Redis Server and Cluster.

Godis

license Build Status Coverage Status Go Report Card Go Reference
Mentioned in Awesome Go

中文版

Godis is a golang implementation of Redis Server, which intents to provide an example of writing a high concurrent middleware using golang.

Key Features:

  • Support string, list, hash, set, sorted set
  • TTL
  • Publish/Subscribe
  • GEO
  • AOF and AOF Rewrite
  • MULTI Commands Transaction is Atomic and Isolated. If any errors are encountered during execution, godis will rollback the executed commands
  • Server-side Cluster which is transparent to client. You can connect to any node in the cluster to access all data in the cluster.
    • MSET, DEL command is supported and atomically executed in cluster mode
    • Rename, RenameNX command is supported within slot in cluster mode
    • MULTI Commands Transaction is supported within slot in cluster mode
  • Concurrent Core, so you don't have to worry about your commands blocking the server too much.

If you could read Chinese, you can find more details in My Blog.

Get Started

You can get runnable program in the releases of this repository, which supports Linux and Darwin system.

./godis-darwin
./godis-linux

You could use redis-cli or other redis client to connect godis server, which listens on 0.0.0.0:6399 on default mode.

The program will try to read config file path from environment variable CONFIG.

If environment variable is not set, then the program try to read redis.conf in the working directory.

If there is no such file, then the program will run with default config.

cluster mode

Godis can work in cluster mode, please append following lines to redis.conf file

peers localhost:7379,localhost:7389 // other node in cluster
self  localhost:6399 // self address

We provide node1.conf and node2.conf for demonstration. use following command line to start a two-node-cluster:

CONFIG=node1.conf ./godis-darwin &
CONFIG=node2.conf ./godis-darwin &

Connect to a node in the cluster to access all data in the cluster:

redis-cli -p 6399

Supported Commands

See: commands.md

Benchmark

Environment:

Go version:1.16

System: macOS Catalina 10.15.7

CPU: 2.6GHz 6-Core Intel Core i7

Memory: 16 GB 2667 MHz DDR4

Performance report by redis-benchmark:

PING_INLINE: 87260.03 requests per second
PING_BULK: 89206.06 requests per second
SET: 85034.02 requests per second
GET: 87565.68 requests per second
INCR: 91157.70 requests per second
LPUSH: 90334.23 requests per second
RPUSH: 90334.23 requests per second
LPOP: 90334.23 requests per second
RPOP: 90415.91 requests per second
SADD: 90909.09 requests per second
HSET: 84104.29 requests per second
SPOP: 82918.74 requests per second
LPUSH (needed to benchmark LRANGE): 78247.26 requests per second
LRANGE_100 (first 100 elements): 26406.13 requests per second
LRANGE_300 (first 300 elements): 11307.10 requests per second
LRANGE_500 (first 450 elements): 7968.13 requests per second
LRANGE_600 (first 600 elements): 6092.73 requests per second
MSET (10 keys): 65487.89 requests per second

Todo List

  • Multi Command
  • Watch Command and CAS support
  • Stream support
  • RDB file loader
  • Master-Slave mode
  • Sentinel

Read My Code

If you want to read my code in this repository, here is a simple guidance.

  • github.com/hdt3213/godis/cmd: only the entry point
  • github.com/hdt3213/godis/config: config parser
  • github.com/hdt3213/godis/interface: some interface definitions
  • github.com/hdt3213/godis/lib: some utils, such as logger, sync utils and wildcard

I suggest focusing on the following directories:

  • github.com/hdt3213/godis/tcp: the tcp server
  • github.com/hdt3213/godis/redis: the redis protocol parser
  • github.com/hdt3213/godis/datastruct: the implements of data structures
    • dict: a concurrent hash map
    • list: a linked list
    • lock: it is used to lock keys to ensure thread safety
    • set: a hash set based on map
    • sortedset: a sorted set implements based on skiplist
  • github.com/hdt3213/godis: the core of storage engine
    • db.go: the basement of database
    • exec.go: the gateway of database
    • router.go: the command table
    • keys.go: handlers for keys commands
    • string.go: handlers for string commands
    • list.go: handlers for list commands
    • hash.go: handlers for hash commands
    • set.go: handlers for set commands
    • sortedset.go: handlers for sorted set commands
    • pubsub.go: implements of publish / subscribe
    • aof.go: implements of AOF persistence and rewrite
    • geo.go: implements of geography features

License

This project is licensed under the GPL license.

Comments
  • 项目启动错误,报常数int类型溢出

    项目启动错误,报常数int类型溢出

    启动项目的电脑环境

    启动项目电脑系统:windows10 / 64位 go版本:go version go1.17.7 windows/386

    执行的操作

    在clone项目之后,进入到目录cmd命令行,执行以下命令

    go mod tidy
    

    安装依赖之后在目录下执行:

    go run main.go
    

    报以下错误:

    # github.com/hdt3213/rdb/core
    D:\Code\golang\go\pkg\mod\github.com\hdt3213\[email protected]\core\list.go:283:21: constant 4294967295 overflows int
    

    这个问题只在 Windows10 操作系统出现,我另外一台 Ubuntu 20.04 操作系统的电脑项目正常运行。 那请问如何在 Windows10 操作系统电脑上运行项目代码呢?

  • Replication 疑问

    Replication 疑问

    在阅读主从同步相关代码的时候,saleof命令执行后,会异步与master进行连接建立,并且会对master发送 REPLCONF listening-port命令,但是我好像并没有在代码中找到这个REPLCONF的实现。是目前不支持嘛?

    我又用redis-cli直接执行了 REPLCONF 得到: (error) ERR unknown command 'replconf'

    想知道为啥会这样呢?

  • rewriteaof 存在死锁的可能

    rewriteaof 存在死锁的可能

    aof重写步骤为:

    1. 开始rewrite,对应startRewrite方法

      1. 获取写锁,暂停aof写入

      2. 获取aof文件大小

      3. 创建rewriteBuffer channel(带缓冲channel)

      4. 生成临时文件

      5. 释放写锁,恢复aof写入

    2. 读取aof文件内容,加载内容到临时DB对象

    3. 根据临时DB对象的数据,生成命令写入临时文件

    4. 结束rewrite

      1. 获取写锁,暂停aof写入

      2. 读取rewriteBuffer,写入aof临时文件

      3. 关闭rewriteBuffer,并设置为nil

      4. 重命名临时文件为aof文件

      5. open 新的aof文件,并设置为db.aofFile

      6. 释放写锁,恢复aof写入

    而主程序,在第一步与第四步之间,一直可以写入aof chan,在处理aof chan中的数据时,同步写入一份到rewriteBuffer chan中,此时会存在一个问题: 程序写入aof时命令较多,超过了rewriteBuffer的缓冲大小,此时会出现 handleAof方法获取到了读锁,但是在写入rewriteBuffer时,阻塞住了,无法释放读锁 image

    而 finishRewrite 方法,在结束rewrite时,需要先获取到写锁,才会接收 rewriteBuffer chan的数据,就会出现锁已被 handleAof占用,finishRewrite方法获取不到锁的情况,从而导致死锁

    image

  • fix: random number & dead-loop problems & perf: slightly optimize the value`s type of map

    fix: random number & dead-loop problems & perf: slightly optimize the value`s type of map

    fix: random number & dead-loop problems & perf: slightly optimize the value`s type of map

    • problem random: uses the provided seed value to initialize the function rand, for fixing the generation of the same int in the case of multiple executions.
    • problem dead-loop: in the case of the whole dict has few data(ex. 2 items), the function "RandomKeys" or "RandomDistinctKeys" is executed. When "RandomKeys" or "RandomDistinctKeys" has not completed execution and all data is cleared, it will be stuck in an endless loop. I`m not sure whether my solution meets your expectation, it needs you to judge, if not I will close this PR. My solution is as follow: when the number of cycles is an integer multiple of "limit * 2", recheck whether "limit >= dict.Len()"

    perf: slightly optimize the value`s type of map to save memory space at datastruct/dict/concurrent.go

    Have a nice day~

    Limited ability, make it better~

  • question about pipeline

    question about pipeline

    i read from here https://www.cnblogs.com/Finley/p/14028402.html i am confused that i think pipeline seems doesn't work that way.

    1. pipeline mode should send many command in one time, it should be not only asynchronous but also batch operation
    2. pipeline mode should be started by user explicitly not by default
  • refactor: replace

    refactor: replace "if else" by "switch" && compact code "if else"

    refactor: replace "if else" by "switch" && compact code "if else"

    • replace "if else" by "switch" in the file cluster/cluster.go
    • compact code "if else" in the file redis/client/client.go

    Limited ability, make it better~

  • single_db.go 里的事务逻辑

    single_db.go 里的事务逻辑

    // Exec executes command within one database
    func (db *DB) Exec(c redis.Connection, cmdLine [][]byte) redis.Reply {
    	// transaction control commands and other commands which cannot execute within transaction
    	cmdName := strings.ToLower(string(cmdLine[0]))
    	if cmdName == "multi" {
    		if len(cmdLine) != 1 {
    			return protocol.MakeArgNumErrReply(cmdName)
    		}
    		return StartMulti(c)
    	} else if cmdName == "discard" {
    		if len(cmdLine) != 1 {
    			return protocol.MakeArgNumErrReply(cmdName)
    		}
    		return DiscardMulti(c)
    	} else if cmdName == "exec" {
    		if len(cmdLine) != 1 {
    			return protocol.MakeArgNumErrReply(cmdName)
    		}
    		return execMulti(db, c)
    	} else if cmdName == "watch" {
    		if !validateArity(-2, cmdLine) {
    			return protocol.MakeArgNumErrReply(cmdName)
    		}
    		return Watch(db, c, cmdLine[1:])
    	}
    	if c != nil && c.InMultiState() {
    		EnqueueCmd(c, cmdLine)
    		return protocol.MakeQueuedReply()
    	}
    
    	return db.execNormalCommand(cmdLine)
    }
    

    在开启事务后,后续的命令貌似不管语法出错还是正确都是返回的protocol.MakeQueuedReply()

    // EnqueueCmd puts command line into `multi` pending queue
    func EnqueueCmd(conn redis.Connection, cmdLine [][]byte) redis.Reply {
    	cmdName := strings.ToLower(string(cmdLine[0]))
    	cmd, ok := cmdTable[cmdName]
    	if !ok {
    		return protocol.MakeErrReply("ERR unknown command '" + cmdName + "'")
    	}
    	if cmd.prepare == nil {
    		return protocol.MakeErrReply("ERR command '" + cmdName + "' cannot be used in MULTI")
    	}
    	if !validateArity(cmd.arity, cmdLine) {
    		// difference with redis: we won't enqueue command line with wrong arity
    		return protocol.MakeArgNumErrReply(cmdName)
    	}
    	conn.EnqueueCmd(cmdLine)
    	return protocol.MakeQueuedReply()
    }
    

    区别在于如果命令不被识别或者参数个数不对的话是不会被加入的conn里的queue的 所以如果命令有语法错误的话,是不会终止整个事务的,只是后续在遇到exec命令的时候不会被执行? 如果这个事务中的其他的命令在运行时没有错误的话,这个事务就变成可以正常提交,那感觉好像不太合理?

    // ExecMulti executes multi commands transaction Atomically and Isolated
    func (db *DB) ExecMulti(conn redis.Connection, watching map[string]uint32, cmdLines []CmdLine) redis.Reply {
    	// prepare
    	writeKeys := make([]string, 0) // may contains duplicate
    	readKeys := make([]string, 0)
    	for _, cmdLine := range cmdLines {
    		cmdName := strings.ToLower(string(cmdLine[0]))
    		cmd := cmdTable[cmdName]
    		prepare := cmd.prepare
    		write, read := prepare(cmdLine[1:])
    		writeKeys = append(writeKeys, write...)
    		readKeys = append(readKeys, read...)
    	}
    	// set watch
    	watchingKeys := make([]string, 0, len(watching))
    	for key := range watching {
    		watchingKeys = append(watchingKeys, key)
    	}
    	readKeys = append(readKeys, watchingKeys...)
    	db.RWLocks(writeKeys, readKeys)
    	defer db.RWUnLocks(writeKeys, readKeys)
    
    	if isWatchingChanged(db, watching) { // watching keys changed, abort
    		return protocol.MakeEmptyMultiBulkReply()
    	}
    	// execute
    	results := make([]redis.Reply, 0, len(cmdLines))
    	aborted := false
    	undoCmdLines := make([][]CmdLine, 0, len(cmdLines))
    	for _, cmdLine := range cmdLines {
    		undoCmdLines = append(undoCmdLines, db.GetUndoLogs(cmdLine))
    		result := db.execWithLock(cmdLine)
    		if protocol.IsErrorReply(result) {
    			aborted = true
    			// don't rollback failed commands
    			undoCmdLines = undoCmdLines[:len(undoCmdLines)-1]
    			break
    		}
    		results = append(results, result)
    	}
    	if !aborted { //success
    		db.addVersion(writeKeys...)
    		return protocol.MakeMultiRawReply(results)
    	}
    	// undo if aborted
    	size := len(undoCmdLines)
    	for i := size - 1; i >= 0; i-- {
    		curCmdLines := undoCmdLines[i]
    		if len(curCmdLines) == 0 {
    			continue
    		}
    		for _, cmdLine := range curCmdLines {
    			db.execWithLock(cmdLine)
    		}
    	}
    	return protocol.MakeErrReply("EXECABORT Transaction discarded because of previous errors.")
    
  • [疑问] 关于同步关闭每个客户端连接的疑问

    [疑问] 关于同步关闭每个客户端连接的疑问

    你好,看到项目中 这个Echohandler Close 有点疑惑,他的本意应该是调用该函数时,关闭所有Conn 并给每个 Conn 一个优雅关闭的超时时间(10s),但在实现上,关闭每个Conn时都是阻塞住,这样子有两个疑问:

    1. 优雅关闭的本意是每个连接给一个超时时间,但 同步去关闭每个Conn,后续关闭的Conn 超过10s了?
    2. 在遍历sync.Map的时候,钩子函数都要阻塞10s(极端情况),这样子 Sync.Map这段时间,其他协程访问它时,均被阻塞了? https://github.com/HDT3213/godis/blob/1459f3de013ead725691a849688089ae4f34a3c9/tcp/echo.go#L82-L86

    感谢~

  • connection/conn.go 这里Write加锁是什么原因呢

    connection/conn.go 这里Write加锁是什么原因呢

    // Write sends response to client over tcp connection
    func (c *Connection) Write(b []byte) error {
    	if len(b) == 0 {
    		return nil
    	}
    	c.mu.Lock()
    	c.waitingReply.Add(1)
    	defer func() {
    		c.waitingReply.Done()
    		c.mu.Unlock()
    	}()
    
    	_, err := c.conn.Write(b)
    	return err
    }
  • perf: slightly optimize config/config.go & fix: channel`s closing parser/parser.go

    perf: slightly optimize config/config.go & fix: channel`s closing parser/parser.go

    perf: slightly optimize config/config.go

    • Exclude the struct tag is space
    • Exclude configuration file comment, which is starting with space

    fix: in the case of panic unexpected happened in the method parse0 of parser/parser.go, closing of the channel is needed to prevent the program from getting stuck~

    Limited ability, hope to make it better~

  • 不会自动生成持久化存储文件吗?

    不会自动生成持久化存储文件吗?

    版本:发布版本1.2.8 redis.conf配置如下 ` bind 0.0.0.0 port 6399 maxclients 128

    appendonly no appendfilename appendonly.aof dbfilename test.rdb ` 当前目录没有生成 appendonly.aof 和 test.rdb 文件

  • rewrite.go FinishRewrite Function Error

    rewrite.go FinishRewrite Function Error

    当我启动aof功能,并执行rewriteaof命令时,aof重写会因为rename失败而失败 原因如下: image 我通过打断点发现rename处无法执行

    同时,观察err发现,tmpAof文件直接写到了C盘的Temp目录下,而不是在项目路径下 我使用的是golang1.19,原本创建tmpFile的命令被遗弃了,新的命令直接是创建在C盘缓存下的 image

    我修改了一下代码,让tmpFile文件直接放在项目路径下:

    	tmpFile, err := os.Create("tmpAofFile.aof")
    	if err != nil {
    		logger.Warn("create temp file failed")
    		return nil, err
    	}
    

    但是依旧还是会出现无法rename的问题:

    [WARN][rewrite.go:169] 2022/11/24 17:14:07 [update aof file failed: rename tmpAofFile.aof appendonly.aof: The process cannot access th
    e file because it is being used by another process.]
    
    
Golang client for redislabs' ReJSON module with support for multilple redis clients (redigo, go-redis)

Go-ReJSON - a golang client for ReJSON (a JSON data type for Redis) Go-ReJSON is a Go client for ReJSON Redis Module. ReJSON is a Redis module that im

Dec 25, 2022
Bxd redis benchmark - Redis benchmark tool for golang

使用 redis benchmark 工具, 测试 10 20 50 100 200 1k 5k 字节 value 大小,redis get set 性能。 r

Jan 22, 2022
Redis client Mock Provide mock test for redis query

Redis client Mock Provide mock test for redis query, Compatible with github.com/go-redis/redis/v8 Install Confirm that you are using redis.Client the

Dec 27, 2022
redis protocol server for go.

redigosrv what redis server侧的golang版本实现。 why 作为后端RD,你开发的,维护的最多的应该就是你的server,作为服务的提供方来说,想要和客户端进行有效的交互,首先要要能理解信息的含义,因此一套通信协议是必须的。 为什么选择redis协议,应用层有很多成熟的

Nov 30, 2021
Implemented Cassandra DB to GoLang API
Implemented Cassandra DB to GoLang API

Implemented Cassandra DB to GoLang API

Jun 7, 2022
Simple key-value store abstraction and implementations for Go (Redis, Consul, etcd, bbolt, BadgerDB, LevelDB, Memcached, DynamoDB, S3, PostgreSQL, MongoDB, CockroachDB and many more)

gokv Simple key-value store abstraction and implementations for Go Contents Features Simple interface Implementations Value types Marshal formats Road

Dec 24, 2022
Examples and code to assign a name to your MongoDB, MySQL, PostgreSQL, RabbitMQ, and redis connection.
Examples and code to assign a name to your MongoDB, MySQL, PostgreSQL, RabbitMQ, and redis connection.

your connection deserves a name ?? When your app interacts with an external system, assign a name to the connection. An external system in this contex

Dec 14, 2022
redis client implement by golang, inspired by jedis.

godis redis client implement by golang, refers to jedis. this library implements most of redis command, include normal redis command, cluster command,

Dec 6, 2022
Type-safe Redis client for Golang

Redis client for Golang ❤️ Uptrace.dev - distributed traces, logs, and errors in one place Join Discord to ask questions. Documentation Reference Exam

Jan 1, 2023
Type-safe Redis client for Golang

Redis client for Golang ❤️ Uptrace.dev - distributed traces, logs, and errors in one place Join Discord to ask questions. Documentation Reference Exam

Jan 4, 2023
Redis client for Golang
Redis client for Golang

Redis client for Golang To ask questions, join Discord or use Discussions. Newsl

Dec 23, 2021
Redis client for Golang
Redis client for Golang

Redis client for Golang Discussions. Newsletter to get latest updates. Documentation Reference Examples RealWorld example app Other projects you may l

Dec 30, 2021
A golang tool to view Redis data in terminal
A golang tool to view Redis data in terminal

Redis Viewer A tool to view Redis data in terminal. Usage: KeyBoard Description ctrl+c exit redis viewer ↑ previous key ↓ next key ← previous page → n

Dec 26, 2022
Google Go Client and Connectors for Redis

Go-Redis Go Clients and Connectors for Redis. The initial release provides the interface and implementation supporting the (~) full set of current Red

Oct 25, 2022
Redis powered simple OTS service - contains two endpoints, to create and find a secret

Onetimesecret This is a simple service that stores and finds your secret. Small but powerfull service - does not have any unnesseccery dependencies. H

Aug 20, 2022
Redis-benchmark - Simple get, mget and pipelined get benchmark.

redis-benchmark Simple get, mget and pipelined get benchmark. Usage git clone https://github.com/Ali-A-A/redis-benchmark.git cd ./redis-benchmark dock

Dec 31, 2021
Priority queue with message-group based partitioning and equal attention guarantee for each message group based on Redis

redis-ordered-queue-go Priority queue with message-group based partitioning and equal attention guarantee for each message group based on Redis What i

Oct 21, 2022
High-performance framework for building redis-protocol compatible TCP servers/services

Redeo The high-performance Swiss Army Knife for building redis-protocol compatible servers/services. Parts This repository is organised into multiple

Jan 4, 2023
Go client for Redis

Redigo Redigo is a Go client for the Redis database. Features A Print-like API with support for all Redis commands. Pipelining, including pipelined tr

Jan 1, 2023