溫馨提示×
您好，登錄后才能下訂單哦！
點(diǎn)擊 登錄注冊 即表示同意《億速云用戶服務(wù)條款》
您好，登錄后才能下訂單哦！
本文小編為大家詳細(xì)介紹“docker如何搭建kafka集群”，內(nèi)容詳細(xì)，步驟清晰，細(xì)節(jié)處理妥當(dāng)，希望這篇“docker如何搭建kafka集群”文章能幫助大家解決疑惑，下面跟著小編的思路慢慢深入，一起來學(xué)習(xí)新知識吧。
1. 刪除所有dangling數(shù)據(jù)卷（即無用的Volume，僵尸文件）
docker volume rm $(docker volume ls -qf dangling=true)
2. 刪除所有dangling鏡像（即無tag的鏡像）
docker rmi $(docker images | grep "^<none>" | awk "{print $3}"3. 刪除所有關(guān)閉的容器
docker ps -a | grep Exit | cut -d ' ' -f 1 | xargs docker rm
環(huán)境為M1版本的mbp:
Zookeeper采用zookeeper
Kafka采用wurstmeister/kafka
Kafka-Manager采用scjtqs/kafka-manager
Mysql采用mysql/mysql-server
1. 新建docker網(wǎng)絡(luò)
docker network create docker-net --subnet 172.20.10.0/16
docker network ls
2. 集群規(guī)劃
| hostname | Ip addr | port | listener |
|---|---|---|---|
| zook1 | 172.20.10.11 | 2184:2181 | |
| zook2 | 172.20.10.12 | 2185:2181 | |
| zook3 | 172.20.10.13 | 2186:2181 | |
| kafka1 | 172.20.10.14 | 內(nèi)部9092:9092，外部9192:9192 | kafka1 |
| kafka2 | 172.20.10.15 | 內(nèi)部9093:9093，外部9193:9193 | kafka2 |
| Kafka3 | 172.20.10.16 | 內(nèi)部9094:9094，外部9194:9194 | Kafka3 |
| 本機(jī)（宿主機(jī)Mbp） | 172.20.10.2 | ||
| kafka manager | 172.20.10.10 | 9000:9000 |
1. 新建文件zk-docker-compose.yml
version: '3.4' services: zook1: image: zookeeper:latest restart: always hostname: zook1 container_name: zook1 #容器名稱，方便在rancher中顯示有意義的名稱 ports: - 2183:2181 #將本容器的zookeeper默認(rèn)端口號映射出去 volumes: # 掛載數(shù)據(jù)卷 - "/Users/konsy/Development/volume/zkcluster/zook1/data:/data" - "/Users/konsy/Development/volume/zkcluster/zook1/datalog:/datalog" - "/Users/konsy/Development/volume/zkcluster/zook1/logs:/logs" environment: ZOO_MY_ID: 1 #即是zookeeper的節(jié)點(diǎn)值，也是kafka的brokerid值 ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181 networks: docker-net: ipv4_address: 172.20.10.11 zook2: image: zookeeper:latest restart: always hostname: zook2 container_name: zook2 #容器名稱，方便在rancher中顯示有意義的名稱 ports: - 2184:2181 #將本容器的zookeeper默認(rèn)端口號映射出去 volumes: - "/Users/konsy/Development/volume/zkcluster/zook2/data:/data" - "/Users/konsy/Development/volume/zkcluster/zook2/datalog:/datalog" - "/Users/konsy/Development/volume/zkcluster/zook2/logs:/logs" environment: ZOO_MY_ID: 2 #即是zookeeper的節(jié)點(diǎn)值，也是kafka的brokerid值 ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181 networks: docker-net: ipv4_address: 172.20.10.12 zook3: image: zookeeper:latest restart: always hostname: zook3 container_name: zook3 #容器名稱，方便在rancher中顯示有意義的名稱 ports: - 2185:2181 #將本容器的zookeeper默認(rèn)端口號映射出去 volumes: - "/Users/konsy/Development/volume/zkcluster/zook3/data:/data" - "/Users/konsy/Development/volume/zkcluster/zook3/datalog:/datalog" - "/Users/konsy/Development/volume/zkcluster/zook3/logs:/logs" environment: ZOO_MY_ID: 3 #即是zookeeper的節(jié)點(diǎn)值，也是kafka的brokerid值 ZOO_SERVERS: server.1=zook1:2888:3888;2181 server.2=zook2:2888:3888;2181 server.3=zook3:2888:3888;2181 networks: docker-net: ipv4_address: 172.20.10.13 networks: docker-net: external: name: docker-net
2. 執(zhí)行腳本部署zookeeper至Docker:
docker compose -f ./zk-docker-compose.yml up -d
1. 新建文件kafka-docker-compose.yml
version: '2' services: kafka1: image: docker.io/wurstmeister/kafka restart: always hostname: kafka1 container_name: kafka1 ports: - 9093:9093 - 9193:9193 environment: KAFKA_BROKER_ID: 1 KAFKA_LISTENERS: INSIDE://:9093,OUTSIDE://:9193 #KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094 SKAFKA_ADVERTISED_LISTENERS: INSIDE://kafka1:9093,OUTSIDE://localhost:9193 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181 ALLOW_PLAINTEXT_LISTENER : 'yes' JMX_PORT: 9999 #開放JMX監(jiān)控端口，來監(jiān)測集群數(shù)據(jù) volumes: - /Users/konsy/Development/volume/kafka/kafka1/wurstmeister/kafka:/wurstmeister/kafka - /Users/konsy/Development/volume/kafka/kafka1/kafka:/kafka external_links: - zook1 - zook2 - zook3 networks: docker-net: ipv4_address: 172.20.10.14 kafka2: image: docker.io/wurstmeister/kafka restart: always hostname: kafka2 container_name: kafka2 ports: - 9094:9094 - 9194:9194 environment: KAFKA_BROKER_ID: 2 KAFKA_LISTENERS: INSIDE://:9094,OUTSIDE://:9194 #KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094 KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka2:9094,OUTSIDE://localhost:9194 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181 ALLOW_PLAINTEXT_LISTENER : 'yes' JMX_PORT: 9999 #開放JMX監(jiān)控端口，來監(jiān)測集群數(shù)據(jù) volumes: - /Users/konsy/Development/volume/kafka/kafka2/wurstmeister/kafka:/wurstmeister/kafka - /Users/konsy/Development/volume/kafka/kafka2/kafka:/kafka external_links: - zook1 - zook2 - zook3 networks: docker-net: ipv4_address: 172.20.10.15 kafka3: image: docker.io/wurstmeister/kafka restart: always hostname: kafka3 container_name: kafka3 ports: - 9095:9095 - 9195:9195 environment: KAFKA_BROKER_ID: 3 KAFKA_LISTENERS: INSIDE://:9095,OUTSIDE://:9195 #KAFKA_ADVERTISED_LISTENERS=INSIDE://<container>:9092,OUTSIDE://<host>:9094 KAFKA_ADVERTISED_LISTENERS: INSIDE://kafka3:9095,OUTSIDE://localhost:9195 KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT KAFKA_INTER_BROKER_LISTENER_NAME: INSIDE KAFKA_ZOOKEEPER_CONNECT: zook1:2181,zook2:2181,zook3:2181 ALLOW_PLAINTEXT_LISTENER : 'yes' JMX_PORT: 9999 #開放JMX監(jiān)控端口，來監(jiān)測集群數(shù)據(jù) volumes: - /Users/konsy/Development/volume/kafka/kafka3/wurstmeister/kafka:/wurstmeister/kafka - /Users/konsy/Development/volume/kafka/kafka3/kafka:/kafka external_links: - zook1 - zook2 - zook3 networks: docker-net: ipv4_address: 172.20.10.16 networks: docker-net: external: name: docker-net
2. 執(zhí)行腳本部署kafka至Docker:
docker compose -f ./kafka-docker-compose.yml up -d
3. listeners 和 advertised.listeners
listeners: 學(xué)名叫監(jiān)聽器，其實(shí)就是告訴外部連接者要通過什么協(xié)議訪問指定主機(jī)名和端口開放的 Kafka 服務(wù)。
advertised.listeners：和 listeners 相比多了個 advertised。Advertised 的含義表示宣稱的、公布的，就是說這組監(jiān)聽器是 Broker 用于對外發(fā)布的。
比如說：
listeners: INSIDE://172.17.0.10:9092,OUTSIDE://172.17.0.10:9094 advertised_listeners: INSIDE://172.17.0.10:9092,OUTSIDE://<公網(wǎng) ip>:端口 kafka_listener_security_protocol_map: "INSIDE:SASL_PLAINTEXT,OUTSIDE:SASL_PLAINTEXT" kafka_inter_broker_listener_name: "INSIDE"
advertised_listeners 監(jiān)聽器會注冊在 zookeeper 中；
當(dāng)我們對 172.17.0.10:9092 請求建立連接，kafka 服務(wù)器會通過 zookeeper 中注冊的監(jiān)聽器，找到 INSIDE 監(jiān)聽器，然后通過 listeners 中找到對應(yīng)的 通訊 ip 和 端口；
同理，當(dāng)我們對 <公網(wǎng) ip>:端口 請求建立連接，kafka 服務(wù)器會通過 zookeeper 中注冊的監(jiān)聽器，找到 OUTSIDE 監(jiān)聽器，然后通過 listeners 中找到對應(yīng)的 通訊 ip 和 端口 172.17.0.10:9094；
總結(jié)：advertised_listeners 是對外暴露的服務(wù)端口，真正建立連接用的是 listeners。
4.安裝kafka-manager
4.1 新建文件kafka-manager-docker-compose.yml
version: '2' services: kafka-manager: image: scjtqs/kafka-manager:latest restart: always hostname: kafka-manager container_name: kafka-manager ports: - 9000:9000 external_links: # 連接本compose文件以外的container - zook1 - zook2 - zook3 - kafka1 - kafka2 - kafka3 environment: ZK_HOSTS: zook1:2181,zook2:2181,zook3:2181 KAFKA_BROKERS: kafka1:9093,kafka2:9094,kafka3:9095 APPLICATION_SECRET: letmein KM_ARGS: -Djava.net.preferIPv4Stack=true networks: docker-net: ipv4_address: 172.20.10.10 networks: docker-net: external: name: docker-net
4.2 執(zhí)行腳本部署kafka-manager至Docker:
docker compose -f ./kafka-manager-docker-compose.yml up -d
4.3 配置Cluster
5. 測試kafka
5.1連接容器
進(jìn)入kafka容器的命令行：
docker exec -ti kafka /bin/bash
進(jìn)入kafka所在目錄：
cd opt/kafka_2.13-2.8.1/
5.2 創(chuàng)建Replication為2，Partition為2的topic
bin/kafka-topics.sh --create --zookeeper zook1:2181 --replication-factor 2 --partitions 2 --topic partopic
5.3 查看topic的狀態(tài)
在kafka容器中的 opt/kafka_2.12-1.1.0/ 目錄下輸入
bin/kafka-topics.sh --describe --zookeeper zook1:2181 --topic partopic
讀到這里，這篇“docker如何搭建kafka集群”文章已經(jīng)介紹完畢，想要掌握這篇文章的知識點(diǎn)還需要大家自己動手實(shí)踐使用過才能領(lǐng)會，如果想了解更多相關(guān)內(nèi)容的文章，歡迎關(guān)注億速云行業(yè)資訊頻道。
免責(zé)聲明：本站發(fā)布的內(nèi)容（圖片、視頻和文字）以原創(chuàng)、轉(zhuǎn)載和分享為主，文章觀點(diǎn)不代表本網(wǎng)站立場，如果涉及侵權(quán)請聯(lián)系站長郵箱：is@yisu.com進(jìn)行舉報，并提供相關(guān)證據(jù)，一經(jīng)查實(shí)，將立刻刪除涉嫌侵權(quán)內(nèi)容。
億速云公眾號
手機(jī)網(wǎng)站二維碼