Rump - Hot sync two Redis servers using dumps.

• [x] I have read the contributing guide.
• [x] Issue: https://github.com/stickermule/rump/issues/41

Implementation

• radix version is upgraded to v3.7.1 and tested in real environment.
• added auth field to redis Resource struct.
• added new function for retrieving redis url and auth token in connection string. It's done for avoiding code duplication.
• added new test case for checking how validation works with auth token.
• when connecting to redis with auth token we need to define callback to perform authentication and encryption as described here. Here is implementation
• Here I check if auth is enabled we use ConnFunc described above, or your method without authentication (same for target).

Thanks!

I've just tried to make a run between SRC=a regular redis and DEST=an elasticache server.

• Rump detects the keys as I see progress on the screen, proportionate to the keys
• but when I look at the keys in the DEST elasticache I only see 1) "ElastiCacheMasterReplicationTimestamp"

Is this supported?

The elasticserver info:

# Server
redis_version:2.8.24
redis_git_sha1:0
redis_git_dirty:0
redis_build_id:0
redis_mode:standalone
os:Amazon ElastiCache
arch_bits:64

I've used this tool it in the past and it works great, but now I have a case in which I need to dump keys that only match a certain pattern instead of dumping all the keys of a DB.

MATCH option of SCAN command can be used to add this support.

The following commit https://github.com/fzunino/rump/commit/67bc2e1dac6d46019019a8f6cf6996ba8a02cf70 adds this support adding an optional command line argument called match and using * as default value, maintaining the current semantic.

Let me know and I can submit a PR with this commit.

Hello.

First of all, thank you very much for this. Very cool.

I noticed that there's no test coverage and, if it's okay, I'd like to add/increase that? I'm on vacation for the next fortnight, but I can get a PR in before the end of the month?

Anthony.

Similar to what the dd does when the user sends the USR1 signal, for windows platform, this is a noop.

Also added the same report of restored keys count to be printed at the end of the sync.

• [ ] I have read the contributing guide.

Report

I have been using RUMP last few months to Sync a primary redis endpoint with a failover (in the even of DR). Ive noticed that RUMP is actually copying keys across and not deleted keys that have been removed from primary. For example primary endpoint has around 166 thousand keys, while the failover now has 26 million. is there a flag i am missing to do this or is it not a-part of the functionality? Thank you

• [x] I have read the contributing guide.

Report

Sync with ttl terminated by 'ERR Invalid TTL value, must be >= 0' The source AWS ElastiCache redis version is 3.2.10 rump -from redis://redis-002.ruf311.0001.cnw1.cache.amazonaws.com.cn:6379/0 -to redis://127.0.0.1:6379/0 -ttl signal: exit

redis read: exit ERR Invalid TTL value, must be >= 0

I just noticed that if dumping TO a cluster with AUTH enabled, and NOT specifying the auth password in the "connection string", rump will not show any error, leading the user to believe that a dump has been performed (when it in fact has not)

The demo link(https://github.com/stickermule/rump#demo) contains an instance of docker image running with rump already installed. However, not able to find an image for the same online

Hello,

rump is a great tool however in certain situations having ttl set to 0 does not work okay (session data etc). May I suggest adding a separate flag for ttl with 0 default, so that default behavior does not change. Please see my local change at https://github.com/stickermule/rump/compare/master...eugenetaranov:ttl, if you find it useful, I'll be happy to make a PR.

In a test environment we did a test transfer between two redis cluster, and we found that there was a bigger number of differences between key numbers. Dest node 1625684 Source node 1673187

How can this happen, or how can we mitigate these issues? How can we debug what cases them? Thank you

• [ ] I have read the contributing guide.

Report

./rump -from /opt/Rump/data/redisu0.rump -to redis://:[email protected]:6379/2 signal: exit w redis write: exit bufio.Scanner: token too long

• [x] I have read the contributing guide.

Report

Please update the docs for the ttl option. It is given without explanation and the example in the README appears incorrect (it lacks a value, which is apparently required).

1. What happens to records with a TTL when the ttl option is not given?
2. What happens when it is given?

• [ yes ] I have read the contributing guide.
• [ yes ] Issue: #42

Implementation

Converted to use net/url for config, cleaned up usage, added tls options

• [x] I have read the contributing guide.

Report

In some cases I need to sync redis where tls is enabled and with auth token. And will be good to have an option to sync redis like this: radix support this option.

rump -from redis://redis-1:6379 -to redis://auth-token-string@redis-2:6379

• [X] I have read the contributing guide.

Report

Question: I have a redis cluster created from the bitnami helm chart here: https://github.com/bitnami/charts/tree/master/bitnami/redis-cluster/#installing-the-chart with 3 master nodes and 3 replica nodes. I am trying to copy data from another redis cluster (with no replicas) into this cluster but I keep getting errors like this:

./rump -from redis://redis-telem-cache-stg-ksat-m-1-redis-cluster:6379 -to redis://10.255.8.12:6379 -ttl
r
signal: exit
rr
redis read: exit
MOVED 4042 10.255.8.14:6379

Note: Above 10.255.8.12:6379 is one of the master nodes.

$ kk exec rails-cache-stg-redis-ksat-m-1-redis-cluster-0 -- redis-cli --cluster check 10.255.8.10:6379
Defaulting container name to rails-cache-stg-redis-ksat-m-1-redis-cluster.
Use 'kubectl describe pod/rails-cache-stg-redis-ksat-m-1-redis-cluster-0 -n default-minim-private' to see all of the containers in this pod.
10.255.8.15:6379 (fbdd5a4e...) -> 14029 keys | 5462 slots | 1 slaves.
10.255.8.12:6379 (b3718ac7...) -> 568 keys | 5461 slots | 1 slaves.
10.255.8.14:6379 (8790c578...) -> 487 keys | 5461 slots | 1 slaves.
[OK] 15084 keys in 3 masters.
0.92 keys per slot on average.
>>> Performing Cluster Check (using node 10.255.8.10:6379)
S: 8500263462d98613bc6e1172106ddcd37f5976a6 10.255.8.10:6379
   slots: (0 slots) slave
   replicates b3718ac766349261d819c9923cd915fc43d70352
M: fbdd5a4eebe77b49ec984ec953ec098fe262d320 10.255.8.15:6379
   slots:[5461-10922] (5462 slots) master
   1 additional replica(s)
S: 4b371b26a9b5b3d0f23d2c965197f0bf3cbc355b 10.255.8.11:6379
   slots: (0 slots) slave
   replicates 8790c57879627039f3d74cc599e61707d33d93d4
M: b3718ac766349261d819c9923cd915fc43d70352 10.255.8.12:6379
   slots:[10923-16383] (5461 slots) master
   1 additional replica(s)
S: f01bd775618fd6de9a18160186a48a37b469da03 10.255.8.13:6379
   slots: (0 slots) slave
   replicates fbdd5a4eebe77b49ec984ec953ec098fe262d320
M: 8790c57879627039f3d74cc599e61707d33d93d4 10.255.8.14:6379
   slots:[0-5460] (5461 slots) master
   1 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

Does rump currently support clusters with replicas? If it does can you recommend a way to diagnose/debug this further?

Thanks.

• [x] I have read the contributing guide.

Report

The current version of rump does not work with an Elasticache Redis cache cluster where in-transit encryption (TLS) is enabled. If you try it, you will only get errors about blocked connections. (More about in-transit encryption: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/in-transit-encryption.html)

Would you be interested in a PR to add support for TLS encrypted connections? I have a private version where this works for my use case. If you are interested in a PR, I could create a more general solution that works with all combinations of encrypted / not encrypted connections and files.

It requires a change to the URL validation, an upgrade of the radix library and some changes to the run package where the connections are managed.