Parallel processing through go routines, copy and delete thousands of key within some minutes

redis-dumper CLI

Parallel processing through go routines, copy and delete thousands of key within some minutes

  • copy data by key pattern from one redis instance to another (backup/restore)
  • delete keys from redis (e.g by key pattern)
  • generate dummy keys

Installation

  • Option A
  • Option B
    • Checkout master and install locally with
    • go install
  • Option C
    • Install with Homebrew
    • brew tap appit-online/redis-dumper https://github.com/appit-online/redis-dumper
      brew install redis-dumper
      

Usage

Source, destination - can be provided as just : or in Redis URL format: redis://[: @] : [/ ] Pattern - can be glob-style pattern supported by Redis SCAN command.

1) Migrate - Export/Import keys

redis-dumper migrate <source> <destination> --pattern="prefix:*" --sourcePassword="SourcePassword" --targetPassword="TargetPassword"

Other flags:

  --sourceUseTLS bool   "Enable TLS for source Redis" (default true)
  --targetUseTLS bool   "Enable TLS for target Redis" (default true)
  --logInterval int     "Print current status every N seconds" (default 1)
  --scanCount int       "COUNT parameter for redis SCAN command" (default 1000)
  --parallelDumps int   "Number of parallel dump goroutines" (default 100)
  --pushRoutines int    "Number of parallel restore goroutines" (default 100)

2) Delete keys

redis-dumper delete <redis> --pattern="prefix:*" --password="Password" 

Other flags:

  --useTLS bool           "Enable TLS" (default true)
  --logInterval int       "Print current status every N seconds" (default 1)
  --scanCount int         "COUNT parameter for redis SCAN command" (default 1000)
  --parallelDeletes int   "Number of parallel delete goroutines" (default 100)

3) Generate keys

redis-dumper generate <redis> --password="Password" 

Other flags:

  --useTLS bool             "Enable TLS" (default true)
  --prefixes []string       "List of prefixes for generated keys" (default {"mykey:", "testkey:"})
  --prefixAmount []string   "Amount of keys to create for each prefix in one iteration" (default {"1", "2"})
  --entryCount int          "Iteration count to perform" (default 1)

Migration Job Details

Scanning

Is performed with a single goroutine, scanned keys are sent to channel

DUMPING

X export goroutines are consuming keys and perform DUMP and PTTL as a pipeline command

Restoring

Results are sent to another channel, where another Y push goroutines are performing RESTORE/REPLACE command on the destination instance

Monitoring

A goroutine outputs status every T seconds

Similar Resources

Copy your HashiCorp Vault secrets to a file

Vault Backup ⚠️ Check the oficial way to backup your HashiCorp Vault. Create a backup file of all HashiCorp Vault kv2 secrets. ./vault-backup -help

Dec 20, 2022

nginx exporter for Prometheus copy code from ingress-nginx use for monit nginx

nginx exporter for Prometheus  copy code from ingress-nginx use for monit nginx

nginx_exporter 介绍 从ingress-nginx官方代码中的expoter迁移出来 用来监控虚拟机上的nginx的expoter 基于官方 controller-v0.49.3 版本移植的代码 软件架构 nginx_socket 通过lua模块monitor.lua 将nginx l

Dec 8, 2021

nginx exporter for Prometheus copy code from ingress-nginx use for monit nginx

nginx exporter for Prometheus copy code from ingress-nginx use for monit nginx

nginx_exporter 介绍 从ingress-nginx官方代码中的expoter迁移出来 用来监控虚拟机上的nginx的expoter 基于官方 controller-v0.49.3 版本移植的代码 软件架构 nginx_socket 通过lua模块monitor.lua 将nginx l

Dec 20, 2021

Coriolis-snapshot-agent - Coriolis snapshot agent leverages the blk-snap kernel module to create copy-on-write snapshots of block devices

Coriolis snapshot agent Coriolis snapshot agent leverages the blk-snap kernel mo

Feb 17, 2022

Fast, Docker-ready image processing server written in Go and libvips, with Thumbor URL syntax

Imagor Imagor is a fast, Docker-ready image processing server written in Go. Imagor uses one of the most efficient image processing library libvips (w

Dec 30, 2022

Telegraf - An agent for collecting, processing, aggregating, and writing metrics

Telegraf Telegraf is an agent for collecting, processing, aggregating, and writi

Feb 11, 2022

Monitor your Website and APIs from your Computer. Get Notified through Slack, E-mail when your server is down or response time is more than expected.

Monitor your Website and APIs from your Computer. Get Notified through Slack, E-mail when your server is down or response time is more than expected.

StatusOK Monitor your Website and APIs from your computer.Get notified through Slack or E-mail when your server is down or response time is more than

Dec 27, 2022

Command-line tool to remotely execute commands on Windows machines through WinRM

winrm-cli This is a Go command-line executable to execute remote commands on Windows machines through the use of WinRM/WinRS. Note: this tool doesn't

Dec 15, 2022

Build a retractable ECS load balance network through aliyun openapi.

ECSEquilizer 通过阿里云OpenAPI建立一个可伸缩的负载均衡网络调度器。 简介 为确保代理集群网络和计算能力可以通过ECS云服务动态伸缩,所以制定实现以下策略。 节点分为static和dynamic两种,static节点是通过配置文件(config.yaml)实现预设的,固定不变。 d

Jul 2, 2022
K8s-delete-protection - Kubernetes admission controller to avoid deleteing master nodes

k8s-delete-protection Admission Controller If you want to make your Kubernetes c

Nov 2, 2022
vcluster - Create fully functional virtual Kubernetes clusters - Each cluster runs inside a Kubernetes namespace and can be started within seconds
vcluster - Create fully functional virtual Kubernetes clusters - Each cluster runs inside a Kubernetes namespace and can be started within seconds

Website • Quickstart • Documentation • Blog • Twitter • Slack vcluster - Virtual Clusters For Kubernetes Lightweight & Low-Overhead - Based on k3s, bu

Jan 4, 2023
Copy files and artifacts via SSH using a binary, docker or Drone CI.

drone-scp Copy files and artifacts via SSH using a binary, docker or Drone CI. Feature Support routines. Support wildcard pattern on source list. Supp

Dec 6, 2022
To copy a secret to another namespace and sync it up-to-date

Secret Mirror Operator This kubebuilder-based Kubernetes operator copies a Secret to another namespace and synchronizes it with the custom resource Se

Jul 20, 2022
Parallel S3 and local filesystem execution tool.
Parallel S3 and local filesystem execution tool.

s5cmd Overview s5cmd is a very fast S3 and local filesystem execution tool. It comes with support for a multitude of operations including tab completi

Jan 5, 2023
kubetnl tunnels TCP connections from within a Kubernetes cluster to a cluster-external endpoint, e.g. to your local machine. (the perfect complement to kubectl port-forward)

kubetnl kubetnl (kube tunnel) is a command line utility to tunnel TCP connections from within a Kubernetes to a cluster-external endpoint, e.g. to you

Dec 16, 2022
An operator to support Haschicorp Vault configuration workflows from within Kubernetes

Vault Config Operator This operator helps set up Vault Configurations. The main intent is to do so such that subsequently pods can consume the secrets

Sep 30, 2022
An operator to support Haschicorp Vault configuration workflows from within Kubernetes
An operator to support Haschicorp Vault configuration workflows from within Kubernetes

Vault Config Operator This operator helps set up Vault Configurations. The main intent is to do so such that subsequently pods can consume the secrets

Dec 19, 2022
Aws asg updater - Update AMIs within AWS Auto Scaling groups automatically.

AWS Auto Scaling Groups Updater AWS Auto Scaling group is a great way of managing Amazon EC2 instances. AWS Auto Scaling group watches the correspondi

Oct 7, 2022
Rotate is a tool for rotating out AWS Auto-Scaling Groups within a k8s cluster

k8s-r8 rotate is a tool for rotating out AWS Auto-Scaling Groups within a k8s cluster. It was developed to make upgrading AMIs as a one command experi

Mar 27, 2022