溫馨提示×
您好，登錄后才能下訂單哦！
點(diǎn)擊 登錄注冊(cè) 即表示同意《億速云用戶服務(wù)條款》
您好，登錄后才能下訂單哦！
HDFS只是Hadoop最基本的一個(gè)服務(wù)，很多其他服務(wù)，都是基于HDFS展開(kāi)的。所以部署一個(gè)HDFS集群，是很核心的一個(gè)動(dòng)作，也是大數(shù)據(jù)平臺(tái)的開(kāi)始。
安裝Hadoop集群，首先需要有Zookeeper才可以完成安裝。如果沒(méi)有Zookeeper，請(qǐng)先部署一套Zookeeper。另外，JDK以及物理主機(jī)的一些設(shè)置等。請(qǐng)參考：
Hadoop集群(一) Zookeeper搭建
Hadoop集群(三) Hbase搭建
Hadoop集群(四) Hadoop升級(jí)
下面開(kāi)始HDFS的安裝
HDFS主機(jī)分配
1 2 3 | 192.168.67.101 c6701 --Namenode+datanode192.168.67.102 c6702 --datanode192.168.67.103 c6703 --datanode |
1. 安裝HDFS，解壓hadoop-2.6.0-EDH-0u2.tar.gz
我同時(shí)下載2.6和2.7版本的軟件，先安裝2.6，然后在執(zhí)行2.6到2.7的升級(jí)步驟
useradd hdfs echo "hdfs:hdfs" | chpasswd su - hdfs cd /tmp/software tar -zxvf hadoop-2.6.0-EDH-0u2.tar.gz -C /home/hdfs/ mkdir -p /data/hadoop/temp mkdir -p /data/hadoop/journal mkdir -p /data/hadoop/hdfs/name mkdir -p /data/hadoop/hdfs/data chown -R hdfs:hdfs /data/hadoop chown -R hdfs:hdfs /data/hadoop/temp chown -R hdfs:hdfs /data/hadoop/journal chown -R hdfs:hdfs /data/hadoop/hdfs/name chown -R hdfs:hdfs /data/hadoop/hdfs/data $ pwd /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop
2. 修改core-site.xml對(duì)應(yīng)的參數(shù)
$ cat core-site.xml <configuration> <!-- 指定hdfs的nameservice為ns --> <property> <name>fs.defaultFS</name> <value>hdfs://ns</value> </property> <!--指定hadoop數(shù)據(jù)臨時(shí)存放目錄--> <property> <name>hadoop.tmp.dir</name> <value>/data/hadoop/temp</value> </property> <property> <name>io.file.buffer.size</name> <value>4096</value> </property> <!--指定zookeeper地址--> <property> <name>ha.zookeeper.quorum</name> <value>c6701:2181,c6702:2181,c6703:2181</value> </property> </configuration>
3. 修改hdfs-site.xml對(duì)應(yīng)的參數(shù)
cat hdfs-site.xml <configuration> <!--指定hdfs的nameservice為ns，需要和core-site.xml中的保持一致,并且ns如果改，整個(gè)文件中，全部的ns要都修改，保持統(tǒng)一 --> <property> <name>dfs.nameservices</name> <value>ns</value> </property> <!-- ns下面有兩個(gè)NameNode，分別是nn1，nn2 --> <property> <name>dfs.ha.namenodes.ns</name> <value>nn1,nn2</value> </property> <!-- nn1的RPC通信地址 --> <property> <name>dfs.namenode.rpc-address.ns.nn1</name> <value>c6701:9000</value> </property> <!-- nn1的http通信地址 --> <property> <name>dfs.namenode.http-address.ns.nn1</name> <value>c6701:50070</value> </property> <!-- nn2的RPC通信地址 --> <property> <name>dfs.namenode.rpc-address.ns.nn2</name> <value>c6702:9000</value> </property> <!-- nn2的http通信地址 --> <property> <name>dfs.namenode.http-address.ns.nn2</name> <value>c6702:50070</value> </property> <!-- 指定NameNode的元數(shù)據(jù)在JournalNode上的存放位置 --> <property> <name>dfs.namenode.shared.edits.dir</name> <value>qjournal://c6701:8485;c6702:8485;c6703:8485/ns</value> </property> <!-- 指定JournalNode在本地磁盤(pán)存放數(shù)據(jù)的位置 --> <property> <name>dfs.journalnode.edits.dir</name> <value>/data/hadoop/journal</value> </property> <!-- 開(kāi)啟NameNode故障時(shí)自動(dòng)切換 --> <property> <name>dfs.ha.automatic-failover.enabled</name> <value>true</value> </property> <!-- 配置失敗自動(dòng)切換實(shí)現(xiàn)方式 --> <property> <name>dfs.client.failover.proxy.provider.ns</name> <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value> </property> <!-- 配置隔離機(jī)制 --> <property> <name>dfs.ha.fencing.methods</name> <value>sshfence</value> </property> <!-- 使用隔離機(jī)制時(shí)需要ssh免登陸 --> <property> <name>dfs.ha.fencing.ssh.private-key-files</name> <value>/home/hdfs/.ssh/id_rsa</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/data/hadoop/hdfs/name</value> </property> <property> <name>dfs.datanode.data.dir</name> <value>/data/hadoop/hdfs/data</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property> <!-- 在NN和DN上開(kāi)啟WebHDFS (REST API)功能,不是必須 --> <property> <name>dfs.webhdfs.enabled</name> <value>true</value> </property> </configuration>
4. 添加slaves文件
$ more slaves c6701 c6702 c6703
--- 安裝C6702的hdfs---
5. 創(chuàng)建c6702的用戶，并為hdfs用戶ssh免密
ssh c6702 "useradd hdfs" ssh c6702 "echo "hdfs:hdfs" | chpasswd" ssh-copy-id hdfs@c6702
6. 拷貝軟件
scp -r /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz root@c6702:/tmp/software/. ssh c6702 "chmod 777 /tmp/software/*"
7. 創(chuàng)建目錄，解壓軟件
ssh hdfs@c6702 "mkdir hdfs" ssh hdfs@c6702 "tar -zxvf /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz -C /home/hdfs" ssh hdfs@c6702 "ls -al hdfs" ssh hdfs@c6702 "ls -al hdfs/hadoop*"
復(fù)制配置文件
ssh hdfs@c6702 "rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml" ssh hdfs@c6702 "rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml" scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves hdfs@c6702:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves
創(chuàng)建hdfs需要的目錄
ssh root@c6702 "mkdir -p /data/hadoop" ssh root@c6702 " chown -R hdfs:hdfs /data/hadoop" ssh hdfs@c6702 "mkdir -p /data/hadoop/temp" ssh hdfs@c6702 "mkdir -p /data/hadoop/journal" ssh hdfs@c6702 "mkdir -p /data/hadoop/hdfs/name" ssh hdfs@c6702 "mkdir -p /data/hadoop/hdfs/data"
--- 安裝C6703的hdfs---
8. 創(chuàng)建c6703的用戶，并為hdfs用戶ssh免密
ssh c6703 "useradd hdfs" ssh c6703 "echo "hdfs:hdfs" | chpasswd" ssh-copy-id hdfs@c6703
9. 拷貝軟件
scp -r /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz root@c6703:/tmp/software/. ssh c6703 "chmod 777 /tmp/software/*" 10. 創(chuàng)建目錄，解壓軟件 ssh hdfs@c6703 "mkdir hdfs" ssh hdfs@c6703 "tar -zxvf /tmp/software/hadoop-2.6.0-EDH-0u2.tar.gz -C /home/hdfs" ssh hdfs@c6703 "ls -al hdfs" ssh hdfs@c6703 "ls -al hdfs/hadoop*"
復(fù)制配置文件
ssh hdfs@c6703 "rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml" ssh hdfs@c6703 "rm -rf /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml" scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/core-site.xml scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/hdfs-site.xml scp -r /home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves hdfs@c6703:/home/hdfs/hadoop-2.6.0-EDH-0u2/etc/hadoop/slaves
創(chuàng)建hdfs需要的目錄
ssh root@c6703 "mkdir -p /data/hadoop" ssh root@c6703 " chown -R hdfs:hdfs /data/hadoop" ssh hdfs@c6703 "mkdir -p /data/hadoop/temp" ssh hdfs@c6703 "mkdir -p /data/hadoop/journal" ssh hdfs@c6703 "mkdir -p /data/hadoop/hdfs/name" ssh hdfs@c6703 "mkdir -p /data/hadoop/hdfs/data"
11. 啟動(dòng)HDFS，先啟動(dòng)三個(gè)節(jié)點(diǎn)的journalnode
/home/hdfs/hadoop-2.6.0-EDH-0u2/sbin/hadoop-daemon.sh start journalnode
檢查狀態(tài)
$ jps 3958 Jps 3868 JournalNode
12. 然后啟動(dòng)namenode,首次啟動(dòng)namenode之前，先在其中一個(gè)節(jié)點(diǎn)(主節(jié)點(diǎn))format namenode信息，信息會(huì)存在于dfs.namenode.name.dir指定的路徑中
<name>dfs.namenode.name.dir</name> <value>/data/hadoop/hdfs/name</value>
$ ./hdfs namenode -format 17/09/26 07:52:17 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = c6701.python279.org/192.168.67.101 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 2.6.0-EDH-0u2 STARTUP_MSG: classpath = /home/hdfs/hadoop-2.6.0-EDHxxxxxxxxxx STARTUP_MSG: build = http://gitlab-xxxxx STARTUP_MSG: java = 1.8.0_144 ************************************************************/ 17/09/26 07:52:17 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 17/09/26 07:52:17 INFO namenode.NameNode: createNameNode [-format] 17/09/26 07:52:18 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration. 17/09/26 07:52:18 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration. Formatting using clusterid: CID-b2f01411-862f-44b2-a6dc-7d17bd48d522 17/09/26 07:52:18 INFO namenode.FSNamesystem: No KeyProvider found. 17/09/26 07:52:18 INFO namenode.FSNamesystem: fsLock is fair:true 17/09/26 07:52:18 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 17/09/26 07:52:18 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true 17/09/26 07:52:18 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000 17/09/26 07:52:18 INFO blockmanagement.BlockManager: The block deletion will start around 2017 Sep 26 07:52:18 17/09/26 07:52:18 INFO util.GSet: Computing capacity for map BlocksMap 17/09/26 07:52:18 INFO util.GSet: VM type = 64-bit 17/09/26 07:52:18 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB 17/09/26 07:52:18 INFO util.GSet: capacity = 2^21 = 2097152 entries 17/09/26 07:52:18 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 17/09/26 07:52:18 INFO blockmanagement.BlockManager: defaultReplication = 2 17/09/26 07:52:18 INFO blockmanagement.BlockManager: maxReplication = 512 17/09/26 07:52:18 INFO blockmanagement.BlockManager: minReplication = 1 17/09/26 07:52:18 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 17/09/26 07:52:18 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false 17/09/26 07:52:18 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 17/09/26 07:52:18 INFO blockmanagement.BlockManager: encryptDataTransfer = false 17/09/26 07:52:18 INFO blockmanagement.BlockManager: maxNumBlocksToLog = 1000 17/09/26 07:52:18 INFO namenode.FSNamesystem: fsOwner = hdfs (auth:SIMPLE) 17/09/26 07:52:18 INFO namenode.FSNamesystem: supergroup = supergroup 17/09/26 07:52:18 INFO namenode.FSNamesystem: isPermissionEnabled = true 17/09/26 07:52:18 INFO namenode.FSNamesystem: Determined nameservice ID: ns 17/09/26 07:52:18 INFO namenode.FSNamesystem: HA Enabled: true 17/09/26 07:52:18 INFO namenode.FSNamesystem: Append Enabled: true 17/09/26 07:52:18 INFO util.GSet: Computing capacity for map INodeMap 17/09/26 07:52:18 INFO util.GSet: VM type = 64-bit 17/09/26 07:52:18 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB 17/09/26 07:52:18 INFO util.GSet: capacity = 2^20 = 1048576 entries 17/09/26 07:52:18 INFO namenode.NameNode: Caching file names occuring more than 10 times 17/09/26 07:52:18 INFO util.GSet: Computing capacity for map cachedBlocks 17/09/26 07:52:18 INFO util.GSet: VM type = 64-bit 17/09/26 07:52:18 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB 17/09/26 07:52:18 INFO util.GSet: capacity = 2^18 = 262144 entries 17/09/26 07:52:18 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 17/09/26 07:52:18 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 17/09/26 07:52:18 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 17/09/26 07:52:18 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 17/09/26 07:52:18 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 17/09/26 07:52:18 INFO util.GSet: Computing capacity for map NameNodeRetryCache 17/09/26 07:52:18 INFO util.GSet: VM type = 64-bit 17/09/26 07:52:18 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB 17/09/26 07:52:18 INFO util.GSet: capacity = 2^15 = 32768 entries 17/09/26 07:52:18 INFO namenode.NNConf: ACLs enabled? false 17/09/26 07:52:18 INFO namenode.NNConf: XAttrs enabled? true 17/09/26 07:52:18 INFO namenode.NNConf: Maximum size of an xattr: 16384 17/09/26 07:52:19 INFO namenode.FSImage: Allocated new BlockPoolId: BP-144216011-192.168.67.101-1506412339757 17/09/26 07:52:19 INFO common.Storage: Storage directory /data/hadoop/hdfs/name has been successfully formatted. 17/09/26 07:52:20 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 17/09/26 07:52:20 INFO util.ExitUtil: Exiting with status 0 17/09/26 07:52:20 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at c6701.python279.org/192.168.67.101 ************************************************************/
13. standby namenode需要先執(zhí)行bootstrapstandby，輸出如下
[hdfs@c6702 sbin]$ ../bin/hdfs namenode -bootstrapstandby 17/09/26 09:44:58 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = c6702.python279.org/192.168.67.102 STARTUP_MSG: args = [-bootstrapstandby] STARTUP_MSG: version = 2.6.0-EDH-0u2 STARTUP_MSG: classpath = /home/hdfs/haxxx STARTUP_MSG: build = http://gitlab-xxxx STARTUP_MSG: java = 1.8.0_144 ************************************************************/ 17/09/26 09:44:58 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] 17/09/26 09:44:58 INFO namenode.NameNode: createNameNode [-bootstrapstandby] 17/09/26 09:44:59 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration. 17/09/26 09:44:59 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration. ===================================================== About to bootstrap Standby ID nn2 from: Nameservice ID: ns Other Namenode ID: nn1 Other NN's HTTP address: http://c6701:50070 Other NN's IPC address: c6701/192.168.67.101:9000 Namespace ID: 793662207 Block pool ID: BP-144216011-192.168.67.101-1506412339757 Cluster ID: CID-b2f01411-862f-44b2-a6dc-7d17bd48d522 Layout version: -60 ===================================================== Re-format filesystem in Storage Directory /data/hadoop/hdfs/name ? (Y or N) y 17/09/26 09:45:16 INFO common.Storage: Storage directory /data/hadoop/hdfs/name has been successfully formatted. 17/09/26 09:45:16 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration. 17/09/26 09:45:16 WARN common.Util: Path /data/hadoop/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration. 17/09/26 09:45:17 INFO namenode.TransferFsImage: Opening connection to http://c6701:50070/imagetransfer?getimage=1&txid=0&storageInfo=-60:793662207:0:CID-b2f01411-862f-44b2-a6dc-7d17bd48d522 17/09/26 09:45:17 INFO namenode.TransferFsImage: Image Transfer timeout configured to 60000 milliseconds 17/09/26 09:45:17 INFO namenode.TransferFsImage: Transfer took 0.01s at 0.00 KB/s 17/09/26 09:45:17 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 size 351 bytes. 17/09/26 09:45:17 INFO util.ExitUtil: Exiting with status 0 17/09/26 09:45:17 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at c6702.python279.org/192.168.67.102 ************************************************************/
14. 檢查狀態(tài)，namenode還沒(méi)有啟動(dòng)
[hdfs@c6702 sbin]$ jps 4539 Jps 3868 JournalNode
15. 啟動(dòng)standby namenode，命令和master啟動(dòng)的方式相同
[hdfs@c6702 sbin]$ ./hadoop-daemon.sh start namenode starting namenode, logging to /home/hdfs/hadoop-2.6.0-EDH-0u2/logs/hadoop-hdfs-namenode-c6702.python279.org.out
16. 再次檢查，namenode已經(jīng)啟動(dòng)
[hdfs@c6702 sbin]$ jps 4640 Jps 4570 NameNode 3868 JournalNode
17. 格式化zkfc,讓在zookeeper中生成ha節(jié)點(diǎn)，在master上執(zhí)行如下命令，完成格式化
[hdfs@c6701 bin]$ ./hdfs zkfc -formatZK 17/09/26 09:59:20 INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at c6701/192.168.67.101:9000 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:host.name=c6701.python279.org 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_144 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/local/jdk1.8.0_144/jre 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.class.path=/home/hdfs/hadoop-2.6.0-EDH-0u2/exxxx 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.library.path=/home/hdfs/hadoop-2.6.0-EDH-0u2/lib/native 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:java.compiler=<NA> 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:os.name=Linux 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:os.arch=amd64 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:os.version=2.6.32-573.el6.x86_64 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:user.name=hdfs 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:user.home=/home/hdfs 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Client environment:user.dir=/home/hdfs/hadoop-2.6.0-EDH-0u2/bin 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=c6701:2181,c6702:2181,c6703:2181 sessionTimeout=5000 watcher=org.apache.hadoop.ha.ActiveStandbyElector$WatcherWithClientRef@20deea7f 17/09/26 09:59:20 INFO zookeeper.ClientCnxn: Opening socket connection to server c6703.python279.org/192.168.67.103:2181. Will not attempt to authenticate using SASL (unknown error) 17/09/26 09:59:20 INFO zookeeper.ClientCnxn: Socket connection established to c6703.python279.org/192.168.67.103:2181, initiating session 17/09/26 09:59:20 INFO zookeeper.ClientCnxn: Session establishment complete on server c6703.python279.org/192.168.67.103:2181, sessionid = 0x35ebc5163710000, negotiated timeout = 5000 17/09/26 09:59:20 INFO ha.ActiveStandbyElector: Session connected. 17/09/26 09:59:20 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/ns in ZK. 17/09/26 09:59:20 INFO zookeeper.ZooKeeper: Session: 0x35ebc5163710000 closed 17/09/26 09:59:20 INFO zookeeper.ClientCnxn: EventThread shut down
18. 格式化完成的檢查
格式成功后，查看zookeeper中可以看到 <<<<<<<<<<<命令沒(méi)確認(rèn)
[zk: localhost:2181(CONNECTED) 1] ls /hadoop-ha
19. 啟動(dòng)zkfc，這個(gè)就是為namenode使用的
./hadoop-daemon.sh start zkfc starting zkfc, logging to /home/hdfs/hadoop-2.6.0-EDH-0u2/logs/hadoop-hdfs-zkfc-c6701.python279.org.out $ jps 4272 DataNode 4402 JournalNode 6339 Jps 6277 DFSZKFailoverController 4952 NameNode
20. 另一個(gè)節(jié)點(diǎn)啟動(dòng)zkfc，
ssh hdfs@c6702 /home/hdfs/hadoop-2.6.0-EDH-0u2/sbin/hadoop-daemon.sh start zkfc $ jps 4981 Jps 4935 DFSZKFailoverController 4570 NameNode 3868 JournalNode
21. 注意：進(jìn)行初始化的時(shí)候，必須保證zk集群已經(jīng)啟動(dòng)了。
1、在ZK中創(chuàng)建znode來(lái)存儲(chǔ)automatic Failover的數(shù)據(jù)，任選一個(gè)NN執(zhí)行完成即可：
sh bin/hdfs zkfc -formatZK
2、啟動(dòng)zkfs，在所有的NN節(jié)點(diǎn)中執(zhí)行以下命令：
sh sbin/hadoop-daemon.sh start zkfc
22. 啟動(dòng)datanode
最后啟動(dòng)集群
/home/hdfs/hadoop-2.6.0-EDH-0u2/sbin/hadoop-daemon.sh start zkfc sh sbin/start-dfs.sh
HDFS安裝過(guò)程中的重點(diǎn)，最后在軟件啟動(dòng)過(guò)程中，一些初始化操作，很重要。
1. 啟動(dòng)全部的journalnode
2. 在namenode1上執(zhí)行， hdfs namenode -format
3. 在namenode1上執(zhí)行， 啟動(dòng)namenode1，命令hadoop-daemon.sh start namenode
4. 在namenode2上執(zhí)行, hdfs namenode -bootstrapstandby
5. 在namenode1上執(zhí)行，格式化zkfc,在zookeeper中生成HA節(jié)點(diǎn), hdfs zkfc -formatZK
6. 啟動(dòng)zkfc，hadoop-daemon.sh start zkfc。 有namenode運(yùn)行的節(jié)點(diǎn)，都要啟動(dòng)ZKFC
7. 啟動(dòng) datanode
HDFS只是Hadoop最基本的一個(gè)模塊，這里已經(jīng)安裝完成，可以為后面的Hbase提供服務(wù)了。
免責(zé)聲明：本站發(fā)布的內(nèi)容（圖片、視頻和文字）以原創(chuàng)、轉(zhuǎn)載和分享為主，文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng)，如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱：is@yisu.com進(jìn)行舉報(bào)，并提供相關(guān)證據(jù)，一經(jīng)查實(shí)，將立刻刪除涉嫌侵權(quán)內(nèi)容。
億速云公眾號(hào)
手機(jī)網(wǎng)站二維碼
