溫馨提示×

您好，登錄后才能下訂單哦！

密碼登錄×
登錄注冊(cè)×
其他方式登錄
點(diǎn)擊 登錄注冊(cè) 即表示同意《億速云用戶服務(wù)條款》

ruby-2.4.6和redis-4.1.3.gem環(huán)境下的redis-cluster集群中的bug

發(fā)布時(shí)間：2020-05-06 11:07:34 來源：網(wǎng)絡(luò) 閱讀：2514 作者：wjw555 欄目：系統(tǒng)運(yùn)維

bug場(chǎng)景說明：
ruby-2.4.6和redis-4.1.3.gem環(huán)境下的redis-cluster集群中，在移除redis-cluster集群中含有數(shù)據(jù)的節(jié)點(diǎn)時(shí)出現(xiàn)的bug。下面詳細(xì)說明下bug出現(xiàn)的場(chǎng)景。

redis-cluster移除節(jié)點(diǎn)：
和節(jié)點(diǎn)添加一樣，移除節(jié)點(diǎn)也有移除主節(jié)點(diǎn)，從節(jié)點(diǎn)。

 1、移除主節(jié)點(diǎn)
   移除節(jié)點(diǎn)使用redis-trib的del-node命令，

redis-trib del-node 127.0.0.1:7002 ${node-id}

127.0.0.1:7002是redis-cluster 集群中的任意節(jié)點(diǎn)，node-id為要?jiǎng)h除的主節(jié)點(diǎn)。 和添加節(jié)點(diǎn)不同，移除節(jié)點(diǎn)node-id是必需的，測(cè)試刪除7002主節(jié)點(diǎn)：
實(shí)例：刪除redis-cluster集群節(jié)點(diǎn)f7a95238e3de39b616b93949ec7c9f86d3867d63 對(duì)應(yīng)的實(shí)例：192.168.1.39:1986
這個(gè)節(jié)點(diǎn)正好有數(shù)據(jù)

[root@mysql-redis39 log]# /data/soft/redis-4.0.12/src/redis-trib.rb del-node 192.168.1.39:1986 f7a95238e3de39b616b93949ec7c9f86d3867d63
>>> Removing node f7a95238e3de39b616b93949ec7c9f86d3867d63 from cluster 192.168.1.39:1986
[ERR] Node 192.168.1.39:1986 is not empty! Reshard data away and try again.
[root@mysql-redis39 log]#

刪除節(jié)點(diǎn)失敗，提示節(jié)點(diǎn)中存在數(shù)據(jù)，不能從redis-cluster中刪除,需要將他的數(shù)據(jù)轉(zhuǎn)移出去，也就是和新增主節(jié)點(diǎn)一樣需重新分片.

需要重新分片:(移除192.168.1.39:1986的4096 個(gè)hash slots 到be2a864214a624789748c7f753377638c6f88751 192.168.1.54:1986 這master節(jié)點(diǎn))


 [root@mysql-redis39 log]# /data/soft/redis-4.0.12/src/redis-trib.rb  reshard  192.168.1.39:1986
>>> Performing Cluster Check (using node 192.168.1.39:1986)
M: f7a95238e3de39b616b93949ec7c9f86d3867d63 192.168.1.39:1986
   slots:6827-10922 (4096 slots) master
   1 additional replica(s)
M: be2a864214a624789748c7f753377638c6f88751 192.168.1.54:1986
   slots:0-1364,5461-6826,10923-12287 (4096 slots) master
   2 additional replica(s)
M: 8db27e432c9ef45fd37bba20e8ca7b71556541a0 192.168.1.39:986
   slots:1365-5460 (4096 slots) master
   1 additional replica(s)
M: 0b3963a0be38ea500013ddcaa5a5524801421dd5 192.168.1.182:1986
   slots:12288-16383 (4096 slots) master
   1 additional replica(s)
S: 907fedbbf999084893a9dff000701f6a9a92381a 192.168.1.105:986
   slots: (0 slots) slave
   replicates 0b3963a0be38ea500013ddcaa5a5524801421dd5
S: ae6e9b0139735cb5dbc4ebf3ad6e01b3f5420db5 192.168.1.54:1986
   slots: (0 slots) slave
   replicates be2a864214a624789748c7f753377638c6f88751
S: 0efd850873fb346fc3c273a9d3aeaac0a9e4d4a8 192.168.1.182:986
   slots: (0 slots) slave
   replicates f7a95238e3de39b616b93949ec7c9f86d3867d63
S: 7a778a0fb0acaa000d003008f50af430497218f3 192.168.1.105:1986
   slots: (0 slots) slave
   replicates 8db27e432c9ef45fd37bba20e8ca7b71556541a0
S: 47fe8159fd5be2e74e823e077d9afe0fd77570c4 192.168.1.54:986
   slots: (0 slots) slave
   replicates be2a864214a624789748c7f753377638c6f88751
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? be2a864214a624789748c7f753377638c6f88751

需要移動(dòng)到全部主節(jié)點(diǎn)上還是單個(gè)主節(jié)點(diǎn)：

將4096個(gè)槽點(diǎn)移動(dòng)到192.168.1.54:1986上，填寫192.168.1.39:1986的node id ：f7a95238e3de39b616b93949ec7c9f86d3867d63

How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? be2a864214a624789748c7f753377638c6f88751
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:f7a95238e3de39b616b93949ec7c9f86d3867d63      ###192.168.1.39:1986的node id 
Source node #2:done
...................
...................
 Moving slot 10920 from f7a95238e3de39b616b93949ec7c9f86d3867d63
 Moving slot 10921 from f7a95238e3de39b616b93949ec7c9f86d3867d63
 Moving slot 10922 from f7a95238e3de39b616b93949ec7c9f86d3867d63
Do you want to proceed with the proposed reshard plan (yes/no)?yes

確認(rèn)之后會(huì)一個(gè)一個(gè)將192.168.1.39:1986的卡槽移到到192.168.1.54:1986上


[root@mysql-redis39 ~]# /data/soft/redis-4.0.12/src/redis-trib.rb  reshard  192.168.1.182:1986
>>> Performing Cluster Check (using node 192.168.1.182:1986)
M: 0b3963a0be38e
0013ddcaa5a5524801421dd5 192.168.1.182:1986
   slots:12288-16383 (4096 slots) master
   1 additional replica(s)
S: 0efd850873fb346fc3c273a9d3aeaac0a9e4d4a8 192.168.1.182:986
   slots: (0 slots) slave
   replicates f7a95238e3de39b616b93949ec7c9f86d3867d63
S: 7a778a0fb0acaa000d003008f50af430497218f3 192.168.1.105:1986
   slots: (0 slots) slave
   replicates 8db27e432c9ef45fd37bba20e8ca7b71556541a0
S: ae6e9b0139735cb5dbc4ebf3ad6e01b3f5420db5 192.168.1.54:1986
   slots: (0 slots) slave
   replicates be2a864214a624789748c7f753377638c6f88751
M: f7a95238e3de39b616b93949ec7c9f86d3867d63 192.168.1.39:1986
   slots:8530-10922 (2393 slots) master
   1 additional replica(s)
S: 47fe8159fd5be2e74e823e077d9afe0fd77570c4 192.168.1.54:986
   slots: (0 slots) slave
   replicates be2a864214a624789748c7f753377638c6f88751
S: 907fedbbf999084893a9dff000701f6a9a92381a 192.168.1.105:986
   slots: (0 slots) slave
   replicates 0b3963a0be38ea500013ddcaa5a5524801421dd5
M: 8db27e432c9ef45fd37bba20e8ca7b71556541a0 192.168.1.39:986
   slots:1365-5460 (4096 slots) master
   1 additional replica(s)
M: be2a864214a624789748c7f753377638c6f88751 192.168.1.54:1986
   slots:0-1364,5461-8529,10923-12287 (5799 slots) master
   2 additional replica(s)
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
[WARNING] Node 192.168.1.39:1986 has slots in migrating state (8530).
[WARNING] Node 192.168.1.54:1986 has slots in importing state (8530).
[WARNING] The following slots are open: 8530
>>> Check slots coverage...
[OK] All 16384 slots covered.

如果在遷移過程遇到下面這樣的錯(cuò)誤：
[WARNING] Node 192.168.1.39:1986 has slots in migrating state (8530).
[WARNING] Node 192.168.1.54:1986 has slots in importing state (8530).
[WARNING] The following slots are open: 8530

[root@mysql-redis39 ~]# /usr/local/redis/bin/redis-cli -h 192.168.1.39 -p 1986
192.168.1.39:1986> cluster setslot 8530 stable
OK

[root@mysql-redis39 ~]# /usr/local/redis/bin/redis-cli -h 192.168.1.54 -p 1986
192.168.1.54:1986> cluster setslot 8530 stable
OK
可以考慮使用命令“redis-trib.rb fix 192.168.0.3:1986”嘗試修復(fù)。需要顯示有節(jié)點(diǎn)處于migrating或importing狀態(tài)，可以登錄到相應(yīng)的節(jié)點(diǎn)，使用命令“cluster setslot (8530 stable”修改，參數(shù)8530為問題顯示的slot的ID。

再次執(zhí)行下面的命令：


[root@mysql-redis39 ~]# /data/soft/redis-4.0.12/src/redis-trib.rb reshard  192.168.1.39:1986
>>> Performing Cluster Check (using node 192.168.1.39:1986)
M: f7a95238e3de39b616b93949ec7c9f86d3867d63 192.168.1.39:1986
   slots:8530-10922 (2393 slots) master
   1 additional replica(s)
M: be2a864214a624789748c7f753377638c6f88751 192.168.1.54:1986
   slots:0-1364,5461-8529,10923-12287 (5799 slots) master
   2 additional replica(s)
M: 8db27e432c9ef45fd37bba20e8ca7b71556541a0 192.168.1.39:986
   slots:1365-5460 (4096 slots) master
   1 additional replica(s)
M: 0b3963a0be38ea100013ddcaa5a5524801421dd5 192.168.1.182:1986
   slots:12288-16383 (4096 slots) master
   1 additional replica(s)
S: 907fedbbf999084893a9dff000701f6a9a92381a 192.168.1.105:986
   slots: (0 slots) slave
   replicates 0b3963a0be38ea100013ddcaa5a5524801421dd5
S: ae6e9b0139735cb5dbc4ebf3ad6e01b3f5420db5 192.168.1.54:1986
   slots: (0 slots) slave
   replicates be2a864214a624789748c7f753377638c6f88751
S: 0efd850873fb346fc3c273a9d3aeaac0a9e4d4a8 192.168.1.182:986
   slots: (0 slots) slave
   replicates f7a95238e3de39b616b93949ec7c9f86d3867d63
S: 7a778a0fb0acaa000d003008f50af430497218f3 192.168.1.105:1986
   slots: (0 slots) slave
   replicates 8db27e432c9ef45fd37bba20e8ca7b71556541a0
S: 47fe8159fd5be2e74e823e077d9afe0fd77570c4 192.168.1.54:986
   slots: (0 slots) slave
   replicates be2a864214a624789748c7f753377638c6f88751
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.
How many slots do you want to move (from 1 to 16384)? 4096
What is the receiving node ID? be2a864214a624789748c7f753377638c6f88751
Please enter all the source node IDs.
  Type 'all' to use all the nodes as source nodes for the hash slots.
  Type 'done' once you entered all the source nodes IDs.
Source node #1:f7a95238e3de39b616b93949ec7c9f86d3867d63
Source node #2:done

Do you want to proceed with the proposed reshard plan (yes/no)? yes
Moving slot 8530 from 192.168.1.39:1986 to 192.168.1.54:1986: 
[ERR] Calling MIGRATE: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)

報(bào)錯(cuò)：
[ERR] Calling MIGRATE: ERR Syntax error, try CLIENT (LIST | KILL | GETNAME | SETNAME | PAUSE | REPLY)

參考網(wǎng)上資料：
https://blog.csdn.net/m0_37128231/article/details/80755478
說這個(gè)是redis-cluster集群管理命令：redis-trib.rb的一個(gè)bug
按照網(wǎng)上說的方法，但是結(jié)果還是在報(bào)錯(cuò)，沒得到解決。

于是重新安裝ruby和redis-gem版本：

 [root@mysql-redis39 redis]# rvm install 2.6.3
Searching for binary rubies, this might take some time.
No binary rubies available for: centos/7/x86_64/ruby-2.6.3.
Continuing with compilation. Please read 'rvm help mount' to get more information on binary rubies.
Checking requirements for centos.
Requirements installation successful.
Installing Ruby from source to: /usr/local/rvm/rubies/ruby-2.6.3, this may take a while depending on your cpu(s)...
ruby-2.6.3 - #downloading ruby-2.6.3, this may take a while depending on your connection...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 13.8M  100 13.8M    0     0  27236      0  0:08:52  0:08:52 --:--:-- 32881
ruby-2.6.3 - #extracting ruby-2.6.3 to /usr/local/rvm/src/ruby-2.6.3.....
ruby-2.6.3 - #configuring......................................................................
ruby-2.6.3 - #post-configuration..
ruby-2.6.3 - #compiling...............................................................................................
ruby-2.6.3 - #installing................................
ruby-2.6.3 - #making binaries executable..
ruby-2.6.3 - #downloading rubygems-3.0.6
ruby-2.6.3 - #extracting rubygems-3.0.6......
ruby-2.6.3 - #removing old rubygems........
ruby-2.6.3 - #installing rubygems-3.0.6...............................................
ruby-2.6.3 - #gemset created /usr/local/rvm/gems/ruby-2.6.3@global
ruby-2.6.3 - #importing gemset /usr/local/rvm/gemsets/global.gems................................................................
ruby-2.6.3 - #generating global wrappers.......
ruby-2.6.3 - #gemset created /usr/local/rvm/gems/ruby-2.6.3
ruby-2.6.3 - #importing gemsetfile /usr/local/rvm/gemsets/default.gems evaluated to empty gem list
ruby-2.6.3 - #generating default wrappers.......
ruby-2.6.3 - #adjusting #shebangs for (gem irb erb ri rdoc testrb rake).
Install of ruby-2.6.3 - #complete 
Ruby was built without documentation, to build it run: rvm docs generate-ri
[root@mysql-redis39 redis]#  ruby --version
ruby 2.6.3p62 (2019-04-16 revision 67580) [x86_64-linux]
 [root@mysql-redis39 redis]# gem install redis --version 3.2.1 

Fetching redis-3.2.1.gem

Successfully installed redis-3.2.1
Parsing documentation for redis-3.2.1
Installing ri documentation for redis-3.2.1
Done installing documentation for redis after 0 seconds
1 gem installed

然后再測(cè)試刪除節(jié)點(diǎn)：
刪除節(jié)點(diǎn)注意：

redis-trib.rb del-node host:port node_id
在刪除節(jié)點(diǎn)之前，其對(duì)應(yīng)的槽必須為空，所以，在進(jìn)行節(jié)點(diǎn)刪除動(dòng)作之前，必須使用redis-trib.rb reshard將其遷移出去。
需要注意的是，如果某個(gè)節(jié)點(diǎn)的槽被完全遷移出去，其對(duì)應(yīng)的slave也會(huì)隨著更新，指向遷移的目標(biāo)節(jié)點(diǎn)。


[root@mysql-redis39 ~]# /data/soft/redis-4.0.12/src/redis-trib.rb  del-node 192.168.1.54:1986 a5fc7cdbbde8a09ec88cc5c63f0b4e74c8f2a43b
>>> Removing node a5fc7cdbbde8a09ec88cc5c63f0b4e74c8f2a43b from cluster 192.168.1.54:1986
/usr/local/rvm/gems/ruby-2.6.3/gems/redis-3.2.1/lib/redis/client.rb:443: warning: constant ::Fixnum is deprecated
>>> Sending CLUSTER FORGET messages to the cluster...
>>> SHUTDOWN the node.

[root@mysql-redis39 ~]# /data/soft/redis-4.0.12/src/redis-trib.rb  check 192.168.1.54:1986 
/usr/local/rvm/gems/ruby-2.6.3/gems/redis-3.2.1/lib/redis/client.rb:443: warning: constant ::Fixnum is deprecated
[ERR] Sorry, can't connect to node 192.168.1.54:1986
[root@mysql-redis39 ~]#

到此處已經(jīng)成功的刪除掉了先還有數(shù)據(jù)的redis-cluster中的節(jié)點(diǎn)實(shí)例。解決一開始出現(xiàn)的bug問題

向AI問一下細(xì)節(jié)

免責(zé)聲明：本站發(fā)布的內(nèi)容（圖片、視頻和文字）以原創(chuàng)、轉(zhuǎn)載和分享為主，文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱：is@yisu.com進(jìn)行舉報(bào)，并提供相關(guān)證據(jù)，一經(jīng)查實(shí)，將立刻刪除涉嫌侵權(quán)內(nèi)容。

AI
助
手