您好,登錄后才能下訂單哦!
由于歷史原因,公司的緩存方案使用的是Codis,并且一個大部門公用一個集群,我們計劃廢棄Codis,用Redis原生的集群架構(gòu),為什么要廢棄Codis呢,主要有兩個原因:1、Codis官方已經(jīng)很久沒有更新維護(hù)了,Redis官方版本已經(jīng)迭代到5.x.x了,codis-server還是3.x.x,Redis的一些新特性無法支持;2、基于風(fēng)險均攤、雞蛋不放一個籃子的原則,目前我們這樣的用法違背了這一原則,如果一個集群出問題,那么整個部門的全部服務(wù)都受影響。在前期和業(yè)務(wù)部門調(diào)研的過程中發(fā)現(xiàn),大家用Codis不僅僅是做緩存,有些業(yè)務(wù)場景還當(dāng)儲存用,比如計數(shù)器等;所以我們需要一個數(shù)據(jù)實時遷移方案,這樣業(yè)務(wù)才能無感知的從Codis遷移到Redis。
1、支持從Codis到Redis Cluster做數(shù)據(jù)遷移
2、支持從Codis到哨兵集群做數(shù)據(jù)遷移
3、支持只遷移部分key
4、支持查看遷移進(jìn)度
1、redis-migrate-tool
redis-migrate-tool是唯品會開源的一款Redis異構(gòu)集群之間的數(shù)據(jù)實時遷移工具,不過已經(jīng)有兩年沒有更新了,我個人覺得這是一款比較完善的工具,特別是數(shù)據(jù)校驗,詳細(xì)功能介紹見GitHub:
https://github.com/vipshop/redis-migrate-tool
2、RedisShake
RedisShake是阿里云基于豌豆莢開源的redis-port進(jìn)行二次開發(fā)的一個支持Redis異構(gòu)集群實時同步的工具,它和redis-migrate-tool相比較,我覺得它的優(yōu)點在于支持前綴key的同步,支持多DB同步,而redis-migrate-tool 只能全量同步,并且如果源做了分庫,同步到目標(biāo)Redis的時候都同步到了db0一個庫里面了,這對于做了分庫場景的業(yè)務(wù)是不可行的,關(guān)于RedisShake的詳細(xì)功能介紹見GitHub:
https://github.com/alibaba/RedisShake
3、redis-port
redis-port是豌豆莢當(dāng)年為了讓大家方便從redis遷移到Codis開源的一個Redis數(shù)據(jù)遷移工具,現(xiàn)在也已經(jīng)很久沒更新了,關(guān)于它的功能也用法見GitHub:
https://github.com/CodisLabs/redis-port
分片master | 密碼 | codis版本 | 哨兵地址 | master地址 | master密碼 | 哨兵redis版本 |
---|---|---|---|---|---|---|
192.168.46.150:10379 | xxx | 3.2.4 | 192.168.9.87:6385 | 192.168.9.87:6384 | 123456 | 5.0.2 |
192.168.47.150:10379 | xxx | 3.2.4 | 192.168.9.88:6385 | 192.168.9.87:6384 | 123456 | 5.0.2 |
xxx | 3.2.4 | 192.168.9.89:6385 | 192.168.9.87:6384 | 123456 | 5.0.2 |
分片master | 密碼 | codis版本 | master node | master密碼 | redis cluster版本 |
---|---|---|---|---|---|
192.168.46.150:10379 | xxx | 3.2.4 | 192.168.9.87:6383 | 123456 | 5.0.2 |
192.168.47.150:10379 | xxx | 3.2.4 | 192.168.9.89:6382 | 123456 | 5.0.2 |
xxx | 3.2.4 | 192.168.9.88:6381 | 123456 | 5.0.2 |
按官方文檔進(jìn)行編譯安裝即可
遷移哨兵的配置文件
vim /chj/app/redis-migrate-tool/rmt_sentinel.conf
[source]
type: single
servers :
- 192.168.46.150:10379
- 192.168.47.150:10379
redis_auth: xxx
[target]
type: single
servers:
- 192.168.9.87:6384
redis_auth: 123456
[common]
listen: 0.0.0.0:8888
遷移redis cluster的配置文件
vim /chj/app/redis-migrate-tool/rmt_cluster.conf
[source]
type: single
servers :
- 192.168.46.150:10379
- 192.168.47.150:10379
redis_auth: xxx
[target]
type: redis cluster
servers:
- 192.168.9.87:6383
- 192.168.9.89:6382
- 192.168.9.88:6381
redis_auth: 123456
[common]
listen: 0.0.0.0:8889
cd /chj/app/redis-migrate-tool
#condis遷移數(shù)據(jù)到哨兵集群
./src/redis-migrate-tool -c rmt_sentinel.conf -o rmt.log -d
#condis遷移數(shù)據(jù)到redis cluster
./src/redis-migrate-tool -c rmt_cluster.conf -o rmt_cluster.log -d
cd /chj/app/redis-migrate-tool
[root@devops-template-test redis-migrate-tool]# ./src/redis-migrate-tool -c rmt_sentinel.conf -C "redis_check 60000"
Check job is running...
[2019-06-25 11:12:09.414] rmt_check.c:848 ERROR: key checked failed: check key's value error, value is inconsistent. key(len:17, type:hash): BigData-IpParse:4
Checked keys: 60000
Inconsistent value keys: 1
Inconsistent expire keys : 0
Other check error keys: 0
Checked OK keys: 59999
Check job finished, used 16.622s
PS
1、"-C "redis_check 60000" 指定要執(zhí)行數(shù)據(jù)校驗,60000指的是校驗數(shù)據(jù)的樣本數(shù),默認(rèn)是1000
2、如果有異常,需要確認(rèn)執(zhí)行異常key的情況
total_msgs_outqueue可以判斷是否有oplog在隊列中等待處理,如果total_msgs_outqueue>0,請繼續(xù)等待,直到total_msgs_outqueue=0才能切換
[root@devops-template-test redis-migrate-tool]# redis-cli -h 127.0.0.1 -p 8889 info
Server
version:0.1.0
os:Linux 3.10.0-693.5.2.el7.x86_64 x86_64
multiplexing_api:epoll
gcc_version:4.8.5
process_id:10137
tcp_port:8889
uptime_in_seconds:1201
uptime_in_days:0
config_file:/chj/app/redis-migrate-tool/rmt_cluster.conf
Clients
connected_clients:1
max_clients_limit:100
total_connections_received:1
Memory
mem_allocator:jemalloc-0.0.0
Group
source_nodes_count:2
target_nodes_count:4
Stats
all_rdb_received:1
all_rdb_parsed:1
all_aof_loaded:0
rdb_received_count:2
rdb_parsed_count:2
aof_loaded_count:0
total_msgs_recv:357666
total_msgs_sent:357666
total_net_input_bytes:78804395
total_net_output_bytes:1688068278
total_net_input_bytes_human:75.15M
total_net_output_bytes_human:1.57G
total_mbufs_inqueue:0
total_msgs_outqueue:0
mkdir /chj/app/redis-shake
cd /chj/app/redis-shake
wget https://github.com/alibaba/RedisShake/releases/download/release-v1.6.9-20190624/redis-shake.tar.gz
tar -zxvf redis-shake.tar.gz
在原來的配置文件上修改,只修改下面有注釋的項,其他保持不變
id = redis-shake
log.file = ./redis-shake.log
log.level = info
pid_path =
system_profile = 9310
http_profile = 9320
ncpu = 0
parallel = 32
source.type = cluster #源類型選擇cluster
source.address = 192.168.46.150:10379;192.168.47.150:10379 #codis 分片master的地址
source.password_raw = xxx #codis的密碼
source.auth_type = auth
source.tls_enable = false
target.type = sentinel #目標(biāo)的類型是哨兵
#target.type = cluster #目標(biāo)是redis cluster
target.address = sentinel-zhj2-redis-sentinel-dev-6384@192.168.9.87:6385;192.168.9.88:6385;192.168.9.89:6385 #目標(biāo)哨兵集群的地址
#target.address = 192.168.9.87:6383;192.168.9.89:6382;192.168.9.88:6381 #目標(biāo)redis cluster的地址
target.password_raw = 123456 #目標(biāo)redis的密碼
target.auth_type = auth
target.db = -1
target.tls_enable = false
rdb.input = local
rdb.output = local_dump
rdb.parallel = 0
rdb.special_cloud =
fake_time =
rewrite = true
filter.db = 0 #只同步db0
filter.key =mms;vcc #只同步mms和vcc開頭的key
filter.slot =
filter.lua = false
big_key_threshold = 524288000
psync = false
metric = true
metric.print_log = false
heartbeat.url =
heartbeat.interval = 3
heartbeat.external = test external
heartbeat.network_interface =
sender.size = 104857600
sender.count = 5000
sender.delay_channel_size = 65535
keep_alive = 0
scan.key_number = 50
scan.special_cloud =
scan.key_file =
qps = 200000
replace_hash_tag = false
extra = false
/chj/app/redis-shake/start.sh /chj/app/redis-shake/redis-shake.conf sync
通過比較PullCommandTotal - BypassCommandTotal == PushCommandTotal 確定同步是否完成
curl http://192.168.47.253:9320/metric| python -m json.tool
[
{
"AvgDelay": "0.43 ms",
"BypassCmdCount": 0,
"BypassCmdCountTotal": 0,
"Delay": "null ms",
"Details": null,
"FailCmdCount": 0,
"FailCmdCountTotal": 0,
"FullSyncProgress": 100,
"NetworkFlowTotal": 42006,
"NetworkSpeed": 0,
"ProcessingCmdCount": 0,
"PullCmdCount": 0,
"PullCmdCountTotal": 897,
"PushCmdCount": 0,
"PushCmdCountTotal": 839,
"SenderBufCount": 0,
"SourceAddress": "192.168.46.150:10379",
"SourceDBOffset": 0,
"StartTime": "2019-06-25T17:45:23Z",
"Status": "incr",
"SuccessCmdCount": 0,
"SuccessCmdCountTotal": 839,
"TargetAddress": [
"192.168.9.87:6384"
],
"TargetDBOffset": 0
},
{
"AvgDelay": "0.60 ms",
"BypassCmdCount": 1,
"BypassCmdCountTotal": 4067,
"Delay": "null ms",
"Details": null,
"FailCmdCount": 0,
"FailCmdCountTotal": 0,
"FullSyncProgress": 100,
"NetworkFlowTotal": 37629,
"NetworkSpeed": 0,
"ProcessingCmdCount": 0,
"PullCmdCount": 1,
"PullCmdCountTotal": 5106,
"PushCmdCount": 0,
"PushCmdCountTotal": 333,
"SenderBufCount": 0,
"SourceAddress": "192.168.47.150:10379",
"SourceDBOffset": 0,
"StartTime": "2019-06-25T17:45:23Z",
"Status": "incr",
"SuccessCmdCount": 0,
"SuccessCmdCountTotal": 333,
"TargetAddress": [
"192.168.9.87:6384"
],
"TargetDBOffset": 0
}
]
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點不代表本網(wǎng)站立場,如果涉及侵權(quán)請聯(lián)系站長郵箱:is@yisu.com進(jìn)行舉報,并提供相關(guān)證據(jù),一經(jīng)查實,將立刻刪除涉嫌侵權(quán)內(nèi)容。