kafka中怎么增加topic的備份數(shù)量

今天就跟大家聊聊有關(guān)kafka中怎么增加topic的備份數(shù)量，可能很多人都不太了解，為了讓大家更加了解，小編給大家總結(jié)了以下內(nèi)容，希望大家根據(jù)這篇文章可以有所收獲。

一、困難點(diǎn)

建立topic的時(shí)候，可以通過(guò)指定參數(shù) --replication-factor 設(shè)置備份數(shù)量。但是，一旦完成建立topic，則無(wú)法通過(guò)kafka-topic.sh 或者 命令修改replica數(shù)量。

二、解決辦法

實(shí)際上，我們可以考慮一種 “另類” 的辦法：可以利用 kafka-reassign-partitions.sh 命令對(duì)所有分區(qū)進(jìn)行重新分布，在做分區(qū)重新分布的時(shí)候，通過(guò)增加每個(gè)分區(qū)的replica備份數(shù)量來(lái)達(dá)到目的。

本文將介紹如何利用 kafka-reassign-partitions.sh 命令增加topic的備份數(shù)量。

注意：以下命令使用到的topic名稱、zookeeper的ip和port，需要讀者替換成為實(shí)際集群的參數(shù)。

（假設(shè)kafka集群有4個(gè)broker，id分別為：1001,1002,1003,1004）

2.1、獲取當(dāng)前topic的所有分區(qū)分布在broker的情況

[root@tbds bin]# ./kafka-topics.sh --zookeeper 172.16.32.13:2181 --topic ranger_audits --describe
Topic:ranger_audits     PartitionCount:10       ReplicationFactor:1     Configs:
        Topic: ranger_audits    Partition: 0    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: ranger_audits    Partition: 1    Leader: 1002    Replicas: 1002  Isr: 1002
        Topic: ranger_audits    Partition: 2    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: ranger_audits    Partition: 3    Leader: 1002    Replicas: 1002  Isr: 1002
        Topic: ranger_audits    Partition: 4    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: ranger_audits    Partition: 5    Leader: 1002    Replicas: 1002  Isr: 1002
        Topic: ranger_audits    Partition: 6    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: ranger_audits    Partition: 7    Leader: 1002    Replicas: 1002  Isr: 1002
        Topic: ranger_audits    Partition: 8    Leader: 1001    Replicas: 1001  Isr: 1001
        Topic: ranger_audits    Partition: 9    Leader: 1002    Replicas: 1002  Isr: 1002

可以看出，ranger_audits 這個(gè)topic有10個(gè)分區(qū)，每個(gè)分區(qū)只有一個(gè)feplica備份，分布在1001和1002兩臺(tái)broker上面。

下面我們需要將ranger_audits 的每個(gè)分區(qū)數(shù)據(jù)都增加到2個(gè)replica備份，且分布到4個(gè)broker上面。

2.2、創(chuàng)建增加replica備份數(shù)量的配置文件

（注意：盡量保持topic的原有每個(gè)分區(qū)的主備份不變化。因此，配置文件的每個(gè)分區(qū)的第一個(gè)broker保持不變。）

[root@tbds bin]# vim ../config/increase-replication-factor.json
{"version":1,
"partitions":[
{"topic":"ranger_audits","partition":0,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":1,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":2,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":3,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":4,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":5,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":6,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":7,"replicas":[1002,1004]},
{"topic":"ranger_audits","partition":8,"replicas":[1001,1003]},
{"topic":"ranger_audits","partition":9,"replicas":[1002,1004]}
]}

上面的配置文件說(shuō)明，我們將topic的每個(gè)分區(qū)都增加了一個(gè)replica，且保持每個(gè)分區(qū)原有的主備份所在broker不變化，將每個(gè)分區(qū)新增的replica備份數(shù)據(jù)放到到1003和1004兩個(gè)broker上面。

2.3、開(kāi)始執(zhí)行增加分區(qū)

[root@tbds bin]# ./kafka-reassign-partitions.sh -zookeeper 172.16.32.13:2181 --reassignment-json-file ../config/increase-replication-factor.json --execute
Current partition replica assignment
{"version":1,"partitions":[{"topic":"ranger_audits","partition":3,"replicas":[1002]},{"topic":"ranger_audits","partition":9,"replicas":[1002]},{"topic":"ranger_audits","partition":8,"replicas":[1001]},{"topic":"ranger_audits","partition":1,"replicas":[1002]},{"topic":"ranger_audits","partition":4,"replicas":[1001]},{"topic":"ranger_audits","partition":2,"replicas":[1001]},{"topic":"ranger_audits","partition":5,"replicas":[1002]},{"topic":"ranger_audits","partition":0,"replicas":[1001]},{"topic":"ranger_audits","partition":6,"replicas":[1001]},{"topic":"ranger_audits","partition":7,"replicas":[1002]}]}

Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions 
{"version":1,"partitions":[{"topic":"ranger_audits","partition":0,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":8,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":5,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":2,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":9,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":1,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":3,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":4,"replicas":[1001,1003]},{"topic":"ranger_audits","partition":7,"replicas":[1002,1004]},{"topic":"ranger_audits","partition":6,"replicas":[1001,1003]}]}

2.4、查看執(zhí)行進(jìn)度

[root@tbds bin]# ./kafka-reassign-partitions.sh -zookeeper 172.16.32.13:2181 --reassignment-json-file ../config/increase-replication-factor.json --verify
Status of partition reassignment:
Reassignment of partition [ranger_audits,0] completed successfully
Reassignment of partition [ranger_audits,8] completed successfully
Reassignment of partition [ranger_audits,5] completed successfully
Reassignment of partition [ranger_audits,2] completed successfully
Reassignment of partition [ranger_audits,9] completed successfully
Reassignment of partition [ranger_audits,1] completed successfully
Reassignment of partition [ranger_audits,3] completed successfully
Reassignment of partition [ranger_audits,4] completed successfully
Reassignment of partition [ranger_audits,7] completed successfully
Reassignment of partition [ranger_audits,6] completed successfully

上面顯示增加分區(qū)操作成功

2.5、再次查看topic的情況

[root@tbds bin]# ./kafka-topics.sh --zookeeper 172.16.32.13:2181 --topic ranger_audits --describe
Topic:ranger_audits     PartitionCount:10       ReplicationFactor:2     Configs:
        Topic: ranger_audits    Partition: 0    Leader: 1001    Replicas: 1001,1003     Isr: 1001,1003
        Topic: ranger_audits    Partition: 1    Leader: 1002    Replicas: 1002,1004     Isr: 1002,1004
        Topic: ranger_audits    Partition: 2    Leader: 1001    Replicas: 1001,1003     Isr: 1001,1003
        Topic: ranger_audits    Partition: 3    Leader: 1002    Replicas: 1002,1004     Isr: 1002,1004
        Topic: ranger_audits    Partition: 4    Leader: 1001    Replicas: 1001,1003     Isr: 1001,1003
        Topic: ranger_audits    Partition: 5    Leader: 1002    Replicas: 1002,1004     Isr: 1002,1004
        Topic: ranger_audits    Partition: 6    Leader: 1001    Replicas: 1001,1003     Isr: 1001,1003
        Topic: ranger_audits    Partition: 7    Leader: 1002    Replicas: 1002,1004     Isr: 1002,1004
        Topic: ranger_audits    Partition: 8    Leader: 1001    Replicas: 1001,1003     Isr: 1001,1003
        Topic: ranger_audits    Partition: 9    Leader: 1002    Replicas: 1002,1004     Isr: 1002,1004

從上面可以看出，備份數(shù)量增加成功

三、進(jìn)一步思考

利用上述介紹的辦法，除了可以用來(lái)增加topic的備份數(shù)量之外，還能夠處理以下幾個(gè)場(chǎng)景：

**1、對(duì)topic的所有分區(qū)數(shù)據(jù)進(jìn)行整體遷移。**怎么理解呢？假如集群有N個(gè)broker，后來(lái)新擴(kuò)容M個(gè)broker。由于新擴(kuò)容的broker磁盤都是空的，原有的broker磁盤占用都很滿。那么我們可以利用上述方法，將存儲(chǔ)在原有N個(gè)broker的某些topic整體搬遷到新擴(kuò)容的M個(gè)broker，進(jìn)而實(shí)現(xiàn)kafka集群的整體數(shù)據(jù)均衡。

具體使用方法就是：通過(guò)編寫2.2章節(jié)的配置文件，將topic的所有分區(qū)都配置到新的M個(gè)broker上面去，再執(zhí)行excute，即可完成topic的所有分區(qū)數(shù)據(jù)整體遷移到新擴(kuò)容的M個(gè)broker節(jié)點(diǎn)。

***2、broker壞掉的情況。***導(dǎo)致某些topic的某些分區(qū)的replica數(shù)量減少，可以利用kafka-reassign-partitions.sh增加replica；

***3、kafka 某些broker磁盤占用很滿，某些磁盤占用又很少。***可以利用kafka-reassign-partitions.sh遷移某些topic的分區(qū)數(shù)據(jù)到磁盤占用少的broker，實(shí)現(xiàn)數(shù)據(jù)均衡；

***4、kafka集群擴(kuò)容。***需要把原來(lái)broker的topic數(shù)據(jù)整體遷移到新的broker，合理利用新擴(kuò)容的broker，實(shí)現(xiàn)負(fù)載均衡。

看完上述內(nèi)容，你們對(duì)kafka中怎么增加topic的備份數(shù)量有進(jìn)一步的了解嗎？如果還想了解更多知識(shí)或者相關(guān)內(nèi)容，請(qǐng)關(guān)注億速云行業(yè)資訊頻道，感謝大家的支持。