MySQL to Redis caching made easy

redisql

GoDoc

MySQL to Redis caching made easy

Example Usage

CLI

Installation:

go install github.com/DGKSK8LIFE/redisql/redisql

Configuration:

Create a YAML file with the following structure:

sqluser: 
sqlpassword: 
sqldatabase:
sqltable:
redisaddr:
redispass:
log:

Usage:

redisql copy -config=pathtofile.yml 

Library

Installation:

go get github.com/DGKSK8LIFE/redisql

Usage:

package main

import (
    "github.com/DGKSK8LIFE/redisql"
)

func main() {
	config := redisql.Config{
		SQLUser:     "josh",
		SQLPassword: "joshmark52",
		SQLDatabase: "celebrities",
		SQLTable:    "celebrity",
		RedisAddr:   "localhost:6379",
		RedisPass:   "joshmark52",
		Log:         true,
	}
	err := config.Copy()
	if err != nil {
		panic(err)
	}
}

Current Functionality and Limitations

  • Simple copying of entire MySQL tables to Redis via CLI and Go Module
  • Improved logs (optional CLI output, improved formatting)
  • Support for multiple Redis data types (lists, sets, etc)
  • Support for other SQL servers
Owner
Comments
  • Added logging with levels.

    Added logging with levels.

    Issue #50: reimplement logging.

    • Initialize it by setting the logging level you want displayed.
    • Create Log outputs with a level for when it should be displayed.
    • Set file output.

    I didn't write any logging in the copying part since that will be affected by the Redis Pipeline change.

    I set it to 3 output levels because I thought that seemed reasonable. Another option could be to have different categories, like Reads, Writes, Actions, etc.

  • [FEATURE REQUEST] Add Docker support for redisql so that things like GitHub Actions can be used for CI/CD; and IDE setup is seamless.

    [FEATURE REQUEST] Add Docker support for redisql so that things like GitHub Actions can be used for CI/CD; and IDE setup is seamless.

    Describe the solution you'd like

    Provide a docker compose solution so a maintainer or user can run this box in isolation of needing PostgreSQL/MySQL and Redis to test this.

    Describe alternatives you've considered

    A docker image could be appropriate for testing which combines Go, MySQL, PostgreSQL and/or Redis in one box, however it would require maintenance of upstream Docker images and there's no performance penalty using docker compose, a docker-compose template allows maintainers to just completely retrash/rebuild their development environment by services.

  • [BUG] Forgot to remove

    [BUG] Forgot to remove "Improved Logs" in README roadmap.

    Expected Behavior

    image

    • The highlighted list item should be removed as I've since removed logging entirely from the library.

    Current Behavior

    • The highlighted list item is not removed
  • use redis pipeline to write CopyToString

    use redis pipeline to write CopyToString

    To solve issue #45

    before data pipelining

    > go test -v -db mysql -rows 10000
    Preparing Test...
    === RUN   TestCopyToString
    --- PASS: TestCopyToString (11.00s)
    === RUN   TestCopyToList
    --- PASS: TestCopyToList (1.25s)
    === RUN   TestCopyToHash
    --- PASS: TestCopyToHash (1.25s)
    PASS
    ok      github.com/DGKSK8LIFE/redisql   18.056s
    

    after data pipelining

    > go test -v -db mysql -rows 10000
    Preparing Test...
    === RUN   TestCopyToString
    --- PASS: TestCopyToString (1.31s)
    === RUN   TestCopyToList
    --- PASS: TestCopyToList (1.05s)
    === RUN   TestCopyToHash
    --- PASS: TestCopyToHash (1.15s)
    PASS
    ok      github.com/DGKSK8LIFE/redisql   7.865s
    
  • [BUG] CopyToString is over 100% slower than other methods

    [BUG] CopyToString is over 100% slower than other methods

    Expected Behavior

    Performance in CopyToString should be comparable to that of CopyToList and CopyToHash.

    Current Behavior

    CopyToString is over 100% slower than the aforementioned two.

    Possible Solution

    Because of Redis' data structures, CopyToString (strings) has to write one key per field, compared to CopyToList and CopyToHash that both write one key per row. This is not necessarily something that can be "quick fixed", but with other performance optimizations, especially transactions and pipelining, performance should improve drastically.

    Steps to Reproduce

    1. Run test cases with ~10k rows or so and your preferred DB.
    2. View output:
    ➜  redisql git:(master) go test -v -db mysql -rows 10000 
    Preparing Test...
    === RUN   TestCopyToString
    --- PASS: TestCopyToString (8.43s)
    === RUN   TestCopyToList
    --- PASS: TestCopyToList (0.75s)
    === RUN   TestCopyToHash
    --- PASS: TestCopyToHash (0.92s)
    PASS
    ok      github.com/DGKSK8LIFE/redisql   17.995s
    
POC de caching en Go en utilisant go-redis/cache

Test-web POC de caching en Go en utilisant go-redis/cache, cette lib permet d'avoir un cache local et un cache redis (appel cache local puis cache red

Nov 19, 2021
LFU Redis implements LFU Cache algorithm using Redis as data storage

LFU Redis cache library for Golang LFU Redis implements LFU Cache algorithm using Redis as data storage LFU Redis Package gives you control over Cache

Nov 10, 2022
Concurrency-safe Go caching library with expiration capabilities and access counters

cache2go Concurrency-safe golang caching library with expiration capabilities. Installation Make sure you have a working Go environment (Go 1.2 or hig

Jan 1, 2023
groupcache is a caching and cache-filling library, intended as a replacement for memcached in many cases.

groupcache Summary groupcache is a distributed caching and cache-filling library, intended as a replacement for a pool of memcached nodes in many case

Dec 31, 2022
Multi-level caching service in Go
Multi-level caching service in Go

IgoVIUM Multi-level caching service in Go. Specifically: Distributed in-memory cache (L1) DB-based cache (L2) Long term historization on persistent vo

Nov 9, 2022
API Cache is a simple caching server, using grpc to accept messages.

API Cache is a simple caching server, using grpc to accept messages. It allows to store key-value pairs, where key is string and value is []byte.

Nov 16, 2021
Design and Implement an in-memory caching library for general use

Cache Implementation in GoLang Problem Statement Design and Implement an in-memory caching library for general use. Must Have Support for multiple Sta

Dec 28, 2021
A simple generic in-memory caching layer

sc sc is a simple in-memory caching layer for golang. Usage Wrap your function with sc - it will automatically cache the values for specified amount o

Jul 2, 2022
🧩 Redify is the optimized key-value proxy for quick access and cache of any other database throught Redis and/or HTTP protocol.

Redify (Any database as redis) License Apache 2.0 Redify is the optimized key-value proxy for quick access and cache of any other database throught Re

Sep 25, 2022
Minicache - Distributed cache implemented in Go. Like Redis but simpler.
Minicache - Distributed cache implemented in Go. Like Redis but simpler.

Distributed cache with client-side consistent hashing, distributed leader-elections, and dynamic node discovery. Supports both HTTP/gRPC interfaces secured with mTLS.

Jan 4, 2023
Go-mysql-orm - Golang mysql orm,dedicated to easy use of mysql

golang mysql orm 个人学习项目, 一个易于使用的mysql-orm mapping struct to mysql table golang结构

Jan 7, 2023
Redis-shake is a tool for synchronizing data between two redis databases. Redis-shake是一个用于在两个redis之间同步数据的工具,满足用户非常灵活的同步、迁移需求。
Redis-shake is a tool for synchronizing data between two redis databases. Redis-shake是一个用于在两个redis之间同步数据的工具,满足用户非常灵活的同步、迁移需求。

RedisShake is mainly used to synchronize data from one redis to another. Thanks to the Douyu's WSD team for the support. 中文文档 English tutorial 中文使用文档

Dec 29, 2022
Golang mysql orm, a personal learning project, dedicated to easy use of mysql

golang mysql orm 个人学习项目, 一个易于使用的mysql-orm mapping struct to mysql table golang结构

Dec 30, 2021
Redis caching layer for Cloudflare KV in Golang
Redis caching layer for Cloudflare KV in Golang

Redis caching layer for Cloudflare KV in Golang

Dec 21, 2022
Schema based, typed Redis caching/memoize framework for Go

cacheme - Redis Caching Framework For Go English | 中文 Statically Typed - 100% statically typed using code generation. Scale Efficiently - thundering h

Sep 27, 2022
POC de caching en Go en utilisant go-redis/cache

Test-web POC de caching en Go en utilisant go-redis/cache, cette lib permet d'avoir un cache local et un cache redis (appel cache local puis cache red

Nov 19, 2021
☔🎀 Translation made with simplicity, yet robust. ~ Backend portion of Arisu, made in Go.

☔ Tsubaki Website • Discord • Discussions Translation made with simplicity, yet robust. Made with ?? using TypeScript, React with Next.js. Tsubaki is

Jan 13, 2022
Go MySQL Driver is a MySQL driver for Go's (golang) database/sql package

Go-MySQL-Driver A MySQL-Driver for Go's database/sql package Features Requirements Installation Usage DSN (Data Source Name) Password Protocol Address

Jan 4, 2023
mysql to mysql 轻量级多线程的库表数据同步

goMysqlSync golang mysql to mysql 轻量级多线程库表级数据同步 测试运行 设置当前binlog位置并且开始运行 go run main.go -position mysql-bin.000001 1 1619431429 查询当前binlog位置,参数n为秒数,查询结

Nov 15, 2022
Quick and dirty test to compare MySQL perf between ARM64 & Rosetta MySQL on M1Pro mac

Quick and dirty test to compare MySQL perf between ARM64 & Rosetta MySQL on M1Pro mac

Nov 5, 2021