您好,登錄后才能下訂單哦!
這篇文章主要講解了“怎么用Docker-Compose搭建Spark集群”,文中的講解內(nèi)容簡單清晰,易于學(xué)習(xí)與理解,下面請大家跟著小編的思路慢慢深入,一起來研究和學(xué)習(xí)“怎么用Docker-Compose搭建Spark集群”吧!
對于Spark集群,我們采用一個(gè)mater節(jié)點(diǎn)和兩個(gè)worker節(jié)點(diǎn)進(jìn)行構(gòu)建。其中,所有的work節(jié)點(diǎn)均分配1一個(gè)core和 1GB的內(nèi)存。
Docker鏡像選擇了bitnami/spark的開源鏡像,選擇的spark版本為2.4.3,docker-compose配置如下:
master: image: bitnami/spark:2.4.3 container_name: master user: root environment: - SPARK_MODE=master - SPARK_RPC_AUTHENTICATION_ENABLED=no - SPARK_RPC_ENCRYPTION_ENABLED=no - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no - SPARK_SSL_ENABLED=no ports: - '8080:8080' - '7077:7077' volumes: - ./python:/python worker1: image: bitnami/spark:2.4.3 container_name: worker1 user: root environment: - SPARK_MODE=worker - SPARK_MASTER_URL=spark://master:7077 - SPARK_WORKER_MEMORY=1G - SPARK_WORKER_CORES=1 - SPARK_RPC_AUTHENTICATION_ENABLED=no - SPARK_RPC_ENCRYPTION_ENABLED=no - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no - SPARK_SSL_ENABLED=no worker2: image: bitnami/spark:2.4.3 container_name: worker2 user: root environment: - SPARK_MODE=worker - SPARK_MASTER_URL=spark://master:7077 - SPARK_WORKER_MEMORY=1G - SPARK_WORKER_CORES=1 - SPARK_RPC_AUTHENTICATION_ENABLED=no - SPARK_RPC_ENCRYPTION_ENABLED=no - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no - SPARK_SSL_ENABLED=no
在master節(jié)點(diǎn)中,也映射了一個(gè)/python目錄,用于存放pyspark代碼,方便運(yùn)行。
對于master節(jié)點(diǎn),暴露出7077端口和8080端口分別用于連接spark以及瀏覽器查看spark UI,在spark UI中,集群狀態(tài)如下圖(啟動后):
如果有需要,可以自行添加worker節(jié)點(diǎn),其中可以修改SPARK_WORKER_MEMORY
與SPARK_WORKER_CORES
對節(jié)點(diǎn)分配的資源進(jìn)行修改。
對于該鏡像而言,默認(rèn)exec進(jìn)去是無用戶的,會導(dǎo)致一些安裝命令權(quán)限的不足,無法安裝。例如需要運(yùn)行pyspark,可能需要安裝numpy、pandas等庫,就無法使用pip完成安裝。而通過user: root
就能設(shè)置默認(rèn)用戶為root用戶,避免上述問題。
同上文一樣,在docker-compose.yml的目錄下執(zhí)行docker-compose up -d
命令,就能一鍵構(gòu)建集群(但是如果需要用到numpy等庫,還是需要自己到各節(jié)點(diǎn)內(nèi)進(jìn)行安裝)。
進(jìn)入master節(jié)點(diǎn)執(zhí)行spark-shell
,成功進(jìn)入:
將上文的Hadoop的docker-compose.yml與本次的結(jié)合,得到新的docker-compose.yml:
version: "1.0" services: namenode: image: bde2020/hadoop-namenode:2.0.0-hadoop3.2.1-java8 container_name: namenode ports: - 9870:9870 - 9000:9000 volumes: - ./hadoop/dfs/name:/hadoop/dfs/name - ./input:/input environment: - CLUSTER_NAME=test env_file: - ./hadoop.env datanode: image: bde2020/hadoop-datanode:2.0.0-hadoop3.2.1-java8 container_name: datanode depends_on: - namenode volumes: - ./hadoop/dfs/data:/hadoop/dfs/data environment: SERVICE_PRECONDITION: "namenode:9870" env_file: - ./hadoop.env resourcemanager: image: bde2020/hadoop-resourcemanager:2.0.0-hadoop3.2.1-java8 container_name: resourcemanager environment: SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864" env_file: - ./hadoop.env nodemanager1: image: bde2020/hadoop-nodemanager:2.0.0-hadoop3.2.1-java8 container_name: nodemanager environment: SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864 resourcemanager:8088" env_file: - ./hadoop.env historyserver: image: bde2020/hadoop-historyserver:2.0.0-hadoop3.2.1-java8 container_name: historyserver environment: SERVICE_PRECONDITION: "namenode:9000 namenode:9870 datanode:9864 resourcemanager:8088" volumes: - ./hadoop/yarn/timeline:/hadoop/yarn/timeline env_file: - ./hadoop.env master: image: bitnami/spark:2.4.3-debian-9-r81 container_name: master user: root environment: - SPARK_MODE=master - SPARK_RPC_AUTHENTICATION_ENABLED=no - SPARK_RPC_ENCRYPTION_ENABLED=no - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no - SPARK_SSL_ENABLED=no ports: - '8080:8080' - '7077:7077' volumes: - ./python:/python worker1: image: bitnami/spark:2.4.3-debian-9-r81 container_name: worker1 user: root environment: - SPARK_MODE=worker - SPARK_MASTER_URL=spark://master:7077 - SPARK_WORKER_MEMORY=1G - SPARK_WORKER_CORES=1 - SPARK_RPC_AUTHENTICATION_ENABLED=no - SPARK_RPC_ENCRYPTION_ENABLED=no - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no - SPARK_SSL_ENABLED=no worker2: image: bitnami/spark:2.4.3-debian-9-r81 container_name: worker2 user: root environment: - SPARK_MODE=worker - SPARK_MASTER_URL=spark://master:7077 - SPARK_WORKER_MEMORY=1G - SPARK_WORKER_CORES=1 - SPARK_RPC_AUTHENTICATION_ENABLED=no - SPARK_RPC_ENCRYPTION_ENABLED=no - SPARK_LOCAL_STORAGE_ENCRYPTION_ENABLED=no - SPARK_SSL_ENABLED=no
運(yùn)行集群(還需要一個(gè)hadoop.env文件見上文)長這樣:
通過Docker容器的映射功能,將本地文件與spark集群的master節(jié)點(diǎn)的/python進(jìn)行了文件映射,編寫的pyspark通過映射可與容器中進(jìn)行同步,并通過docker exec指令,完成代碼執(zhí)行:
運(yùn)行了一個(gè)回歸程序,集群功能正常:
感謝各位的閱讀,以上就是“怎么用Docker-Compose搭建Spark集群”的內(nèi)容了,經(jīng)過本文的學(xué)習(xí)后,相信大家對怎么用Docker-Compose搭建Spark集群這一問題有了更深刻的體會,具體使用情況還需要大家實(shí)踐驗(yàn)證。這里是億速云,小編將為大家推送更多相關(guān)知識點(diǎn)的文章,歡迎關(guān)注!
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場,如果涉及侵權(quán)請聯(lián)系站長郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。