大數(shù)據(jù)Hadoop集群搭建

發(fā)布時間：2020-06-30 09:08:45 來源：網(wǎng)絡(luò) 閱讀：2403 作者：898009427 欄目：大數(shù)據(jù)

一、環(huán)境

服務(wù)器配置：

CPU型號：Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
CPU核數(shù)：16
內(nèi)存：64GB

操作系統(tǒng)

版本：CentOS Linux release 7.5.1804 (Core)
主機(jī)列表：

IP	主機(jī)名
192.168.1.101	node1
192.168.1.102	node2
192.168.1.103	node3
192.168.1.104	node4
192.168.1.105	node5

軟件安裝包路徑：/data/tools/
JAVA_HOME路徑：/opt/java # java為軟鏈接，指向jdk的指定版本
Hadoop集群路徑：/data/bigdata/

軟件版本及部署分布：

組件名	安裝包	說明	node1	node2	node3	node4	node5
JDK	jdk-8u162-linux-x64.tar.gz	基礎(chǔ)環(huán)境	?	?	?	?	?
zookeeper	zookeeper-3.4.12.tar.gz		?	?	?
Hadoop	hadoop-2.7.6.tar.gz		?	?	?	?	?
spark	spark-2.1.2-bin-hadoop2.7.tgz		?	?	?	?	?
scala	scala-2.11.12.tgz		?	?	?	?	?
hbase	hbase-1.2.6-bin.tar.gz		?	?	?
hive	apache-hive-2.3.3-bin.tar.gz			?
kylin	apache-kylin-2.3.1-hbase1x-bin.tar.gz			?
kafka	kafka_2.11-1.1.0.tgz					?	?
hue	hue-3.12.0.tgz		?
flume	apache-flume-1.8.0-bin.tar.gz		?	?	?	?	?

注：所有的軟鏈接不可以跨服務(wù)器傳輸，應(yīng)該單獨創(chuàng)建；否則會把軟鏈接所指向的文件或整個目錄傳過去；

二、常用命令

1、查看系統(tǒng)基本配置：

[root@localhost ~]# uname -a
Linux node1 3.10.0-123.9.3.el7.x86_64 #1 SMP Thu Nov 6 15:06:03 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
[root@localhost ~]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
[root@localhost ~]# free -m
             total       used       free     shared    buffers     cached
Mem:              64267          2111        62156         16        212       1190
-/+ buffers/cache:        708      63559
Swap:             32000          0        32000
[root@localhost ~]# lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                16
On-line CPU(s) list:   0-15
Thread(s) per core:    2
Core(s) per socket:    8
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 79
Model name:            Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz
Stepping:              1
CPU MHz:               2095.148
BogoMIPS:              4190.29
Hypervisor vendor:     KVM
Virtualization type:   full
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              20480K
NUMA node0 CPU(s):     0-15
[root@localhost ~]# df -h
文件系統(tǒng)        容量  已用  可用 已用% 掛載點
/dev/sda2       100G  3.1G   97G    4% /
devtmpfs        7.7G     0  7.7G    0% /dev
tmpfs           7.8G     0  7.8G    0% /dev/shm
tmpfs           7.8G  233M  7.5G    3% /run
tmpfs           7.8G     0  7.8G    0% /sys/fs/cgroup
/dev/sda1       500M  9.8M  490M    2% /boot/efi
/dev/sda4       1.8T  9.3G  1.8T    1% /data
tmpfs           1.6G     0  1.6G    0% /run/user/1000

2、啟動集群

start-dfs.sh
start-yarn.sh

3、關(guān)閉集群

stop-yarn.sh
stop-dfs.sh

4、監(jiān)控集群

hdfs dfsadmin -report

5、單個進(jìn)程啟動/關(guān)閉

hadoop-daemon.sh start|stop namenode|datanode| journalnode
yarn-daemon.sh start |stop resourcemanager|nodemanager

三、環(huán)境準(zhǔn)備（所有服務(wù)器）

1、設(shè)置主機(jī)名（其它類似）

[root@localhost ~]# hostnamectl set-hostname node1

2、關(guān)閉防火墻firewalld并禁止開機(jī)自啟動和SELINUX

[root@node1 ~]# systemctl disable firewalld.service 
[root@node1 ~]# systemctl stop firewalld.service 
[root@node1 ~]# systemctl status firewalld.service
    ● firewalld.service - firewalld - dynamic firewall daemon
         Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
            Active: inactive (dead)
         Docs: man:firewalld(1)

[root@node1 ~]# setenforce 0 
[root@node1 ~]# sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config 
[root@node1 ~]# grep SELINUX=disabled /etc/selinux/config
SELINUX=disabled

3、修改hosts文件

[root@node1 ~]# vim /etc/hosts
#127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.1.101 node1
192.168.1.102 node2
192.168.1.103 node3
192.168.1.104 node4
192.168.1.105 node5

4、設(shè)置ssh免密登陸,可用ansible

[root@node1 ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa
[root@node1 ~]# cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
[root@node1 ~]#  ll -d .ssh/
drwx------ 2 root root 4096 Jun  5 08:50 .ssh/
[root@node1 ~]#  ll .ssh/   
total 12
-rw-r--r-- 1 root root 599 Jun  5 08:50 authorized_keys
-rw------- 1 root root 672 Jun  5 08:50 id_dsa
-rw-r--r-- 1 root root 599 Jun  5 08:50 id_dsa.pub

# 把其它服務(wù)器的~/.ssh/id_dsa.pub內(nèi)容也追加到node1服務(wù)器的~/.ssh/authorized_keys文件中，然后分發(fā)
[root@node1 ~]#  scp –rp ~/.ssh/authorized_keys node2: ~/.ssh/
[root@node1 ~]#  scp –rp ~/.ssh/authorized_keys node3: ~/.ssh/
[root@node1 ~]#  scp –rp ~/.ssh/authorized_keys node4: ~/.ssh/
[root@node1 ~]#  scp –rp ~/.ssh/authorized_keys node5: ~/.ssh/

也可以node1生成一套密鑰，然后把~/.ssh整個目錄分發(fā)到其它服務(wù)器，共用一個密鑰

5、修改文件句柄數(shù)

[root@node1 ~]# vim /etc/security/limits.conf
#---------custom-----------------------
#
*           soft   nofile       240000
*           hard   nofile       655350
*           soft   nproc        240000
*           hard   nproc        655350
#-----------end-----------------------
[root@node1 ~]# source /etc/security/limits.conf
[root@node1 ~]# ulimit -n
24000

6、時間同步

ntp服務(wù)器設(shè)置

# 局域網(wǎng)內(nèi)設(shè)置一臺ntp服務(wù)器，其它和這臺ntp同步即可，云服務(wù)器一般默認(rèn)已同步
[root@node1 ~]# yum install ntp -y          # 安裝ntp服務(wù)
[root@node1 ~]# cp -a /etc/ntp.conf{,.bak}
[root@node1 ~]# vim /etc/ntp.conf
restrict default kod nomodify notrap nopeer noquery　 # restrict、default定義默認(rèn)訪問規(guī)則，nomodify禁止遠(yuǎn)程主機(jī)修改本地服務(wù)器
restrict 127.0.0.1　　　　　　　　　　　                # 這里的查詢是服務(wù)器本身狀態(tài)的查詢。
restrict -6 ::1
#server 0.centos.pool.ntp.org iburst          # 注掉官方自帶的網(wǎng)絡(luò)站點
#server 1.centos.pool.ntp.org iburst
#server 2.centos.pool.ntp.org iburst
#server 3.centos.pool.ntp.org iburst
server ntp1.aliyun.com                  # 目標(biāo)服務(wù)器網(wǎng)絡(luò)位置
server 127.127.1.0      # local clock,當(dāng)服務(wù)器與公用的時間服務(wù)器失去聯(lián)系時，就是連不上互聯(lián)網(wǎng)時，以局域網(wǎng)內(nèi)的時間服務(wù)器為客戶端提供時間同步服務(wù)。
fudge  127.127.1.0 stratum 10

# 如果計劃任務(wù)有時間同步，先注釋，兩種用法會沖突。
[root@node1 ~]# crontab –e
#*/30 * * * * /usr/sbin/ntpdate ntp1.aliyun.com > /dev/null 2>&1;/sbin/hwclock -w

# 啟動服務(wù)并設(shè)置開啟自啟:
[root@node1 /]# systemctl start ntpd.service　　 # 啟動服務(wù)
[root@node1 /]# systemctl enable ntpd.service　 # 設(shè)置為開機(jī)啟動
[root@node1 ~]# systemctl status ntpd.service
● ntpd.service - Network Time Service
   Loaded: loaded (/usr/lib/systemd/system/ntpd.service; disabled; vendor preset: disabled)
   Active: active (running) since 一 2018-05-21 13:47:33 CST; 1 weeks 2 days ago
 Main PID: 17915 (ntpd)
   CGroup: /system.slice/ntpd.service
           └─17915 /usr/sbin/ntpd -u ntp:ntp -g

5月 23 11:41:40 node1 ntpd[17915]: Listen normally on 14 enp0s25 192.168.1.101 UDP 123
5月 23 11:41:40 node1 ntpd[17915]: new interface(s) found: waking up resolver
5月 23 11:41:42 node1 ntpd[17915]: Listen normally on 18 enp0s25 fe80::6a85:bbb1:ad57:f6ae UDP 123
[root@node1 ~]# ntpq -p                 # 檢查時間服務(wù)器是否正確同步
     remote           refid      st t when poll reach   delay   offset  jitter
==============================================================================
*time5.aliyun.co 10.137.38.86     2 u  468 1024  377   14.374   -4.292   6.377

當(dāng)所有遠(yuǎn)程服務(wù)器（不是本地服務(wù)器）的jitter值都為4000，并且reach和dalay的值是0時，就表示時間同步有問題?？赡茉蛴?個：
    1）服務(wù)器端的防火墻設(shè)置，阻斷了123端口（可以用 iptables -t filter -A INPUT -p udp --destination-port 123 -j ACCEPT 解決）
　　2）每次重啟ntp服務(wù)器之后，大約3-5分鐘客戶端才能與服務(wù)端建立連接，建立連接之后才能進(jìn)行時間同步，否則客戶端同步時間時會顯示
　　　no server suitable for synchronization found的報錯信息，不用擔(dān)心，等會就可以了。

其它主機(jī)設(shè)置,以node2為例

[root@node2 /]# systemctl stop ntpd.service　　  # 關(guān)閉ntp服務(wù)
[root@node2 /]# systemctl disable ntpd.service　 # 禁止開機(jī)自啟動
[root@node2 ~]# yum install ntpdate -y
[root@node2 ~]# /usr/sbin/ntpdate 192.168.1.101
30 May 17:54:09 ntpdate[20937]: adjust time server 192.168.1.101 offset 0.000758 sec
[root@node2 ~]# crontab –e
*/30 * * * * /usr/sbin/ntpdate 192.168.1.101 > /dev/null 2>&1;/sbin/hwclock -w
[root@node2 ~]# systemctl restart crond.service
[root@node2 ~]# systemctl status crond.service
● crond.service - Command Scheduler
     Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled; vendor preset: enabled)
     Active: active (running) since 四 2018-05-31 09:05:39 CST; 11s ago
     Main PID: 12162 (crond)
     CGroup: /system.slice/crond.service
                     └─12162 /usr/sbin/crond -n

5月 31 09:05:39 node2 systemd[1]: Started Command Scheduler.
5月 31 09:05:39 node2 systemd[1]: Starting Command Scheduler...

7、上傳安裝包到node1服務(wù)器

[root@node1 ~]# mkdir -pv /data/tools
[root@node1 ~]# cd /data/tools
[root@node1 tools]# ll
total 1221212
-rw-r--r-- 1 root root  58688757 May 24 10:23 apache-flume-1.8.0-bin.tar.gz
-rw-r--r-- 1 root root 232229830 May 24 10:25 apache-hive-2.3.3-bin.tar.gz
-rw-r--r-- 1 root root 286104833 May 24 10:26 apache-kylin-2.3.1-hbase1x-bin.tar.gz
-rw-r--r-- 1 root root 216745683 May 24 10:28 hadoop-2.7.6.tar.gz
-rw-r--r-- 1 root root 104659474 May 24 10:27 hbase-1.2.6-bin.tar.gz
-rw-r--r-- 1 root root  47121634 May 24 10:29 hue-3.12.0.tgz
-rw-r--r-- 1 root root  56969154 May 24 10:49 kafka_2.11-1.1.0.tgz
-rw-r--r-- 1 root root 193596110 May 24 10:29 spark-2.1.2-bin-hadoop2.7.tgz
-rw-r--r-- 1 root root  36667596 May 24 10:28 zookeeper-3.4.12.tar.gz
[root@node1 bigdata]#

8、安裝JDK

[root@node1 ~]# tar xf /data/tools/jdk-8u162-linux-x64.tar.gz -C /opt/
[root@node1 ~]# [ -L "/opt/java" ] && rm -f /opt/java
[root@node1 ~]# cd /opt/ && ln -s /opt/jdk1.8.0_162 /opt/java
[root@node1 ~]# chown -R root:root /opt/jdk1.8.0_162
[root@node1 ~]# echo -e "# java\nexport JAVA_HOME=/opt/java\nexport PATH=\${PATH}:\${JAVA_HOME}/bin:\${JAVA_HOME}/jre/bin\nexport CLESSPATH=.:\${JAVA_HOME}/lib:\${JAVA_HOME}/jre/lib" > /etc/profile.d/java_version.sh
[root@node1 ~]# source /etc/profile.d/java_version.sh
[root@node1 ~]# java -version
java version "1.8.0_162"
Java(TM) SE Runtime Environment (build 1.8.0_162-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.162-b12, mixed mode)

四、安裝zookeeper

官方文檔

1、解壓zookeeper

[root@node1 ~]# mkdir -pv /data/bigdata/src
[root@node1 ~]# tar -zxvf  /data/tools/zookeeper-3.4.12.tar.gz  -C /data/bigdata/src
[root@node1 ~]# ln -s /data/bigdata/src/zookeeper-3.4.12  /data/bigdata/zookeeper

# 添加環(huán)境變量
[root@node1 ~]# echo  -e "# zookeeper\nexport ZOOKEEPER_HOME=/data/bigdata/zookeeper\nexport PATH=\$ZOOKEEPER_HOME/bin:\$PATH" > /etc/profile.d/bigdata_path.sh
[root@node1 ~]# cat /etc/profile.d/bigdata_path.sh
# zookeeper
export ZOOKEEPER_HOME=/data/bigdata/zookeeper
export PATH=$ZOOKEEPER_HOME/bin:$PATH
[root@node1 ~]#

2、配置zoo.cfg文件

[root@node1 ~]# cd /data/bigdata/zookeeper/conf/                    #進(jìn)入conf目錄
[root@node1 conf]# cp  zoo_sample.cfg  zoo.cfg                      #拷貝模板
[root@node1 conf]# vim zoo.cfg
# The number of millinode2s of each tick
tickTime=2000
# The number of ticks that the initial 
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between 
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just 
# example sakes.
dataDir=/data/bigdata/zookeeper/data                   #  添加
dataLogDir=/data/bigdata/zookeeper/dataLog       #  添加
# the port at which the clients will connect
clientPort=2181
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the 
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1
server.1=node1:2888:3888     #  添加
server.2=node2:2888:3888
server.3=node3:2888:3888

3、添加myid，分發(fā)(安裝個數(shù)為奇數(shù))

# 創(chuàng)建指定目錄：dataDir目錄下增加myid文件;myid中寫當(dāng)前zookeeper服務(wù)的id, 因為server.1=node1:2888:3888 server指定的是1,
[root@node1 conf]# mkdir  -pv /data/bigdata/zookeeper/{data,dataLog}
[root@node1 conf]# echo 1 > /data/bigdata/zookeeper/data/myid

4、分發(fā)：

[root@node2 ~]# mkdir -pv /data/bigdata/
[root@node3 ~]# mkdir -pv /data/bigdata/

[root@node1 conf]# scp -rp /data/bigdata/src  node2:/data/bigdata/
[root@node1 conf]# scp -rp /data/bigdata/src  node3:/data/bigdata/

[root@node2 ~]# ln -s /data/bigdata/src/zookeeper-3.4.12  /data/bigdata/zookeeper
[root@node3 ~]# ln -s /data/bigdata/src/zookeeper-3.4.12  /data/bigdata/zookeeper

# 在其余機(jī)子配置，node2下面的myid是2，node3下面myid是3，這些都是根據(jù)server來的
[root@node2 ~]# echo 2 > /data/bigdata/zookeeper/data/myid
[root@node3 ~]# echo 3 > /data/bigdata/zookeeper/data/myid

五、安裝Hadoop

官方文檔

生產(chǎn)環(huán)境：兩個主節(jié)點只裝namenode，不裝datanode；

1、解壓hadoop

[root@node1 ~]# tar -zxvf /data/tools/hadoop-2.7.6.tar.gz -C /data/bigdata/src
[root@node1 ~]# ln -s /data/bigdata/src/hadoop-2.7.6 /data/bigdata/hadoop

# 添加環(huán)境變量
[root@node1 ~]# echo  -e "\n# hadoop\nexport HADOOP_HOME=/data/bigdata/hadoop\nexport PATH=\$HADOOP_HOME/bin:\$HADOOP_HOME/sbin\$PATH" >> /etc/profile.d/bigdata_path.sh
[root@node1 ~]# cat /etc/profile.d/bigdata_path.sh
# zookeeper
export ZOOKEEPER_HOME=/data/bigdata/zookeeper
export PATH=$ZOOKEEPER_HOME/bin:$PATH

# hadoop
export HADOOP_HOME=/data/bigdata/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH
[root@node1 ~]#

2、配置hadoop-env.sh

[root@node1 ~]# cd  /data/bigdata/hadoop/etc/hadoop/
[root@node1 hadoop]# vim hadoop-env.sh 
export JAVA_HOME=/opt/java     # 添加
export HADOOP_SSH_OPTS="-p 22"

3、配置core-site.xml

[root@node1 hadoop]# vim  core-site.xml 
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>  
  <!--Yarn 需要使用 fs.defaultFS 指定NameNode URI -->
    <property>  
        <name>fs.defaultFS</name>  
        <value>hdfs://mycluster</value>  
    </property>  

    <property>
        <name>hadoop.tmp.dir</name>
        <value>/data/bigdata/tmp</value>
</property>
<property>
        <name>ha.zookeeper.quorum</name>
        <value>node1:2181,node2:2181,node3:2181</value>
        <discription>zookeeper客戶端連接地址</discription>
</property>

<property>
        <name>ha.zookeeper.session-timeout.ms</name>
        <value>10000</value>
</property>

<property>
        <name>fs.trash.interval</name>
        <value>1440</value>
        <discription>以分鐘為單位的垃圾回收時間，垃圾站中數(shù)據(jù)超過此時間，會被刪除。如果是0，垃圾回收機(jī)制關(guān)閉。</discription>
</property>

<property>
        <name>fs.trash.checkpoint.interval</name>
        <value>1440</value>
        <discription>以分鐘為單位的垃圾回收檢查間隔。</discription>
</property>

<property>
        <name>hadoop.security.authentication</name>
        <value>simple</value>
        <discription>可以設(shè)置的值為 simple (無認(rèn)證) 或者 kerberos（一種安全認(rèn)證系統(tǒng)）</discription>
</property>

    <!-- 與hue集成所需配置-->
    <property>
        <name>hadoop.proxyuser.hue.hosts</name>
        <value>*</value>
    </property>
    <property>
        <name>hadoop.proxyuser.hue.groups</name>
        <value>*</value>
    </property>
    <property>
       <name>hadoop.proxyuser.root.hosts</name>
       <value>*</value>
    </property>
    <property>
       <name>hadoop.proxyuser.root.groups</name>
       <value>*</value>
    </property>
</configuration>

新建指定目錄

[root@node1 hadoop]# mkdir -p /data/bigdata/tmp

4、配置yarn-site.xml

[root@node1 hadoop]# vim yarn-site.xml
<?xml version="1.0"?>
<configuration>
    <property>
        <name>yarn.app.mapreduce.am.scheduler.connection.wait.interval-ms</name>
        <value>5000</value>
        <discription>schelduler失聯(lián)等待連接時間</discription>
    </property>
    <property>
        <name>yarn.nodemanager.aux-services</name>
        <value>mapreduce_shuffle</value>
        <discription>NodeManager上運行的附屬服務(wù)。需配置成mapreduce_shuffle，才可運行MapReduce程序</discription>
    </property>
    <property>
        <name>yarn.resourcemanager.connect.retry-interval.ms</name>
        <value>5000</value>
        <description>How often to try connecting to the ResourceManager.</description>
   </property>
    <property>
        <name>yarn.resourcemanager.ha.enabled</name>
        <value>true</value>
        <discription>是否啟用RM HA，默認(rèn)為false（不啟用）</discription>
    </property>
    <property>
         <name>yarn.resourcemanager.ha.automatic-failover.enabled</name>
         <value>true</value>
         <discription>是否啟用自動故障轉(zhuǎn)移。默認(rèn)情況下，在啟用HA時，啟用自動故障轉(zhuǎn)移。</discription>
   </property>
   <property>
         <name>yarn.resourcemanager.ha.automatic-failover.embedded</name>
        <value>true</value>
         <discription>啟用內(nèi)置的自動故障轉(zhuǎn)移。默認(rèn)情況下，在啟用HA時，啟用內(nèi)置的自動故障轉(zhuǎn)移。</discription>
   </property>
    <property>
        <name>yarn.resourcemanager.cluster-id</name>
        <value>cluster1</value>
        <discription>集群的Id，elector使用該值確保RM不會做為其它集群的active。</discription>
    </property>
    <property>
        <name>yarn.resourcemanager.ha.rm-ids</name>
        <value>rm1,rm2</value>
        <discription>RMs的邏輯id列表,rm管理資源器；一般配兩個，一個起作用  其他備用；用逗號分隔,如:rm1,rm2 </discription>
    </property>
    <property>
        <name>yarn.resourcemanager.hostname.rm1</name>
        <value>node3</value>
        <discription>RM的hostname</discription>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address.rm1</name>
        <value>${yarn.resourcemanager.hostname.rm1}:8030</value>
        <discription>RM對AM暴露的地址,AM通過地址想RM申請資源,釋放資源等</discription>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm1</name>
        <value>${yarn.resourcemanager.hostname.rm1}:8031</value>
        <discription>RM對NM暴露地址,NM通過該地址向RM匯報心跳,領(lǐng)取任務(wù)等</discription>
    </property>
    <property>
        <name>yarn.resourcemanager.address.rm1</name>
        <value>${yarn.resourcemanager.hostname.rm1}:8032</value>
        <discription>RM對客戶端暴露的地址,客戶端通過該地址向RM提交應(yīng)用程序等</discription>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address.rm1</name>
        <value>${yarn.resourcemanager.hostname.rm1}:8033</value>
        <discription>RM對管理員暴露的地址.管理員通過該地址向RM發(fā)送管理命令等</discription>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address.rm1</name>
        <value>${yarn.resourcemanager.hostname.rm1}:8088</value>
        <discription>RM對外暴露的web http地址，用戶可通過該地址在瀏覽器中查看集群信息</discription>
    </property>
    <property>
       <description>The https adddress of the RM web application.</description>
       <name>yarn.resourcemanager.webapp.https.address.rm1</name>
       <value>${yarn.resourcemanager.hostname.rm1}:8090</value>
   </property>
    <property>
        <name>yarn.resourcemanager.hostname.rm2</name>
        <value>node4</value>
    </property>
    <property>
        <name>yarn.resourcemanager.scheduler.address.rm2</name>
        <value>${yarn.resourcemanager.hostname.rm2}:8030</value>
    </property>
    <property>
        <name>yarn.resourcemanager.resource-tracker.address.rm2</name>
        <value>${yarn.resourcemanager.hostname.rm2}:8031</value>
    </property>
    <property>
        <name>yarn.resourcemanager.address.rm2</name>
        <value>${yarn.resourcemanager.hostname.rm2}:8032</value>
    </property>
    <property>
        <name>yarn.resourcemanager.admin.address.rm2</name>
        <value>${yarn.resourcemanager.hostname.rm2}:8033</value>
    </property>
    <property>
        <name>yarn.resourcemanager.webapp.address.rm2</name>
        <value>${yarn.resourcemanager.hostname.rm2}:8088</value>
    </property>
    <property>
       <description>The https adddress of the RM web application.</description>
       <name>yarn.resourcemanager.webapp.https.address.rm2</name>
       <value>${yarn.resourcemanager.hostname.rm2}:8090</value>
   </property>
    <property>
        <name>yarn.resourcemanager.recovery.enabled</name>
        <value>true</value>
        <discription>默認(rèn)值為false，也就是說resourcemanager掛了相應(yīng)的正在運行的任務(wù)在rm恢復(fù)后不能重新啟動</discription>
    </property>
    <property>
        <name>yarn.resourcemanager.store.class</name>
        <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
        <discription>狀態(tài)存儲的類</discription>
    </property>
    <property>
       <name>ha.zookeeper.quorum</name>
       <value>node1:2181,node2:2181,node3:2181</value>
   </property>
    <property>
       <name>yarn.resourcemanager.zk-address</name>
       <value>${ha.zookeeper.quorum}</value>
       <discription>ZooKeeper服務(wù)器的地址（主機(jī)：端口號），既用于狀態(tài)存儲也用于內(nèi)嵌的leader-election。</discription>
   </property>
    <property>
       <name>yarn.nodemanager.address</name>
       <value>${yarn.nodemanager.hostname}:8041</value>
       <discription>The address of the container manager in the NM.</discription>
   </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>58000</value>
        <discription>該節(jié)點上nodemanager可使用的物理內(nèi)存總量</discription>
    </property>
    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>16</value>
        <discription>該節(jié)點上nodemanager可使用的虛擬CPU個數(shù)</discription>
    </property>
    <property>
       <name>yarn.nodemanager.vmem-pmem-ratio</name>
       <value>2</value>
       <discription>任務(wù)每使用1MB物理內(nèi)存，最多可使用虛擬內(nèi)存量，默認(rèn)是2.1。</discription>
   </property>
    <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>1024</value>
        <discription>單個任務(wù)可申請的最小物理內(nèi)存量</discription>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-mb</name>
        <value>58000</value>
        <discription>單個任務(wù)可申請的最大物理內(nèi)存量</discription>
    </property>
    <property>
        <name>yarn.scheduler.minimum-allocation-vcores</name>
        <value>1</value>
        <discription>單個任務(wù)可申請的最小虛擬CPU個數(shù)</discription>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-vcores</name>
        <value>16</value>
        <discription>單個任務(wù)可申請的最大虛擬CPU個數(shù)</discription>
    </property>
</configuration>

5、配置mapred-site.xml

[root@node1 hadoop]# cp mapred-site.xml{.template,}
[root@node1 hadoop]# vim mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
   <property>
       <name>mapreduce.framework.name</name>
       <value>yarn</value>
   </property>

   <property>
       <name>mapreduce.jobhistory.address</name>
       <value>sjfx:10020</value>
   </property>

</configuration>

6、配置hdfs-site.xml

[root@node1 hadoop]# vim hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
        <name>dfs.permissions</name>
        <value>true</value>
        <description>
            If "true", enable permission checking in HDFS.
            If "false", permission checking is turned off,
            but all other behavior is unchanged.
            Switching from one parameter value to the other does not change the mode,
            owner or group of files or directories.
        </description>
    </property>
    <property>
        <name>dfs.replication</name>
        <value>2</value>
        <description>保存副本數(shù)</description>
    </property>

    <property>
        <discription>持久存儲名字空間，事務(wù)日志的本地路徑</discription>
        <name>dfs.namenode.name.dir</name>
        <value>/data/bigdata/hdfs/name</value>
    </property>
    <property>
        <discription>datanode存放數(shù)據(jù)的路徑，單個節(jié)點單配，多個目錄逗號分隔</discription>
        <name>dfs.datanode.data.dir</name>
        <value>/data/bigdata/hdfs/data</value>
    </property>
    <property>
        <discription>指定用于在DataNode間傳輸block數(shù)據(jù)的最大線程數(shù)</discription>
        <name>dfs.datanode.max.transfer.threads</name>
        <value>16384</value>
    </property>
    <property>
        <name>dfs.datanode.balance.bandwidthPerSec</name>
        <value>52428800</value>
        <description>
              Specifies the maximum amount of bandwidth that each datanode
              can utilize for the balancing purpose in term of
              the number of bytes per second.
        </description>
    </property>
    <property>
        <name>dfs.datanode.balance.max.concurrent.moves</name>
        <value>50</value>
        <description>增加DataNode上轉(zhuǎn)移block的Xceiver的個數(shù)上限。</description>
    </property>
    <property>
        <name>dfs.nameservices</name>
        <value>mycluster</value>
    </property>
    <property>
        <name>dfs.ha.namenodes.mycluster</name>
        <value>nn1,nn2</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.mycluster.nn1</name>
        <value>node1:8020</value>
    </property>
    <property>
        <name>dfs.namenode.rpc-address.mycluster.nn2</name>
        <value>node2:8020</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.mycluster.nn1</name>
        <value>node1:50070</value>
    </property>
    <property>
        <name>dfs.namenode.http-address.mycluster.nn2</name>
        <value>node2:50070</value>
    </property>
  <property>
       <name>dfs.namenode.journalnode</name>
       <value>node1:8485;node2:8485;node3:8485</value>
            <discription>journalnode為了解決hadoop單點故障，給namenode做元數(shù)據(jù)同步的，奇數(shù)個,一般3個或5個</discription>
   </property>

    <property>
        <name>dfs.namenode.shared.edits.dir</name>
        <value>qjournal://${dfs.namenode.journalnode}/mycluster</value>
    </property>
    <property>
        <name>dfs.client.failover.proxy.provider.mycluster</name>
        <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
    </property>
    <property>
        <name>dfs.ha.fencing.methods</name>
        <value>sshfence</value>
    </property>
    <property>
        <name>dfs.ha.fencing.ssh.private-key-files</name>
        <value>/root/.ssh/id_dsa</value>
    </property>
    <property>
        <name>dfs.journalnode.edits.dir</name>
        <value>${hadoop.tmp.dir}/dfs/journal</value>
    </property>
    <property>
        <name>dfs.permissions.superusergroup</name>
        <value>root</value>
        <description>超級用戶組名</description>
    </property>

    <property>
        <name>dfs.ha.automatic-failover.enabled</name>
        <value>true</value>
        <description>開啟自動故障轉(zhuǎn)移</description>
    </property>
</configuration>

新建相應(yīng)目錄

[root@node1 hadoop]# mkdir -pv /data/bigdata/tmp/dfs/journal

7、配置capacity-scheduler.xml

[root@node1 hadoop]#  vim capacity-scheduler.xml
<configuration>
    <property>
        <name>yarn.scheduler.capacity.maximum-applications</name>
        <value>10000</value>
        <description>
          Maximum number of applications that can be pending and running.
        </description>
    </property>

    <property>
        <name>yarn.scheduler.capacity.maximum-am-resource-percent</name>
        <value>0.1</value>
        <description>
          Maximum percent of resources in the cluster which can be used to run 
          application masters i.e. controls number of concurrent running
          applications.
        </description>
    </property>

    <property>
        <name>yarn.scheduler.capacity.resource-calculator</name>
        <value>org.apache.hadoop.yarn.util.resource.DominantResourceCalculator</value>
        <description>
          The ResourceCalculator implementation to be used to compare 
          Resources in the scheduler.
          The default i.e. DefaultResourceCalculator only uses Memory while
          DominantResourceCalculator uses dominant-resource to compare 
          multi-dimensional resources such as Memory, CPU etc.
        </description>
    </property>

    <property>
        <name>yarn.scheduler.capacity.root.queues</name>
        <value>default</value>
        <description>
          The queues at the this level (root is the root queue).
        </description>
    </property>

    <property>
        <name>yarn.scheduler.capacity.root.default.capacity</name>
        <value>100</value>
        <description>Default queue target capacity.</description>
    </property>

    <property>
        <name>yarn.scheduler.capacity.root.default.user-limit-factor</name>
        <value>1</value>
        <description>
          Default queue user limit a percentage from 0.0 to 1.0.
        </description>
    </property>

    <property>
        <name>yarn.scheduler.capacity.root.default.maximum-capacity</name>
        <value>100</value>
        <description>
          The maximum capacity of the default queue. 
        </description>
    </property>

    <property>
        <name>yarn.scheduler.capacity.root.default.state</name>
        <value>RUNNING</value>
        <description>
          The state of the default queue. State can be one of RUNNING or STOPPED.
        </description>
    </property>

    <property>
        <name>yarn.scheduler.capacity.root.default.acl_submit_applications</name>
        <value>*</value>
        <description>
          The ACL of who can submit jobs to the default queue.
        </description>
    </property>

    <property>
        <name>yarn.scheduler.capacity.root.default.acl_administer_queue</name>
        <value>*</value>
        <description>
          The ACL of who can administer jobs on the default queue.
        </description>
    </property>

    <property>
        <name>yarn.scheduler.capacity.node-locality-delay</name>
        <value>40</value>
        <description>
          Number of missed scheduling opportunities after which the CapacityScheduler 
          attempts to schedule rack-local containers. 
          Typically this should be set to number of nodes in the cluster, By default is setting 
          approximately number of nodes in one rack which is 40.
        </description>
    </property>

    <property>
        <name>yarn.scheduler.capacity.queue-mappings</name>
        <value></value>
        <description>
          A list of mappings that will be used to assign jobs to queues
          The syntax for this list is [u|g]:[name]:[queue_name][,next mapping]*
          Typically this list will be used to map users to queues,
          for example, u:%user:%user maps all users to queues with the same name
          as the user.
        </description>
    </property>

    <property>
        <name>yarn.scheduler.capacity.queue-mappings-override.enable</name>
        <value>false</value>
        <description>
          If a queue mapping is present, will it override the value specified
          by the user? This can be used by administrators to place jobs in queues
          that are different than the one specified by the user.
          The default is false.
        </description>
    </property>
</configuration>

8、配置slaves

[root@node1 hadoop]# vim  slaves
node1
node2
node3
node4
node5

9、修改$HADOOP_HOME/sbin/hadoop-daemon.sh

[root@node1 hadoop]# cd /data/bigdata/hadoop/sbin/
[root@node1 sbin]# vim hadoop-daemon.sh
#添加：
HADOOP_PID_DIR=/data/bigdata/hdfs/pids
YARN_PID_DIR=/data/bigdata/hdfs/pids

# 新建相應(yīng)目錄
[root@node1 sbin]# mkdir -pv /data/bigdata/hdfs/{name,data,pids}

10、修改$HADOOP_HOME/sbin/yarn-daemon.sh

#添加：
[root@node1 sbin]# vim yarn-daemon.sh
HADOOP_PID_DIR=/data/bigdata/hdfs/pids
YARN_PID_DIR=/data/bigdata/hdfs/pids

11、分發(fā)

[root@node1 sbin]# scp -rp /data/bigdata/src/hadoop-2.7.6  node2:/data/bigdata/src/
[root@node1 sbin]# scp -rp /data/bigdata/src/hadoop-2.7.6  node3:/data/bigdata/src/
[root@node1 sbin]# scp -rp /data/bigdata/src/hadoop-2.7.6  node4:/data/bigdata/src/
[root@node1 sbin]# scp -rp /data/bigdata/src/hadoop-2.7.6  node5:/data/bigdata/src/

[root@node2 ~]# ln -s /data/bigdata/src/hadoop-2.7.6 /data/bigdata/hadoop
[root@node3 ~]# ln -s /data/bigdata/src/hadoop-2.7.6 /data/bigdata/hadoop
[root@node4 ~]# ln -s /data/bigdata/src/hadoop-2.7.6 /data/bigdata/hadoop
[root@node5 ~]# ln -s /data/bigdata/src/hadoop-2.7.6 /data/bigdata/hadoop

六、啟動過程

1、啟動zookeeper服務(wù):下面兩種方法選一

(1) 同時開啟所有zookeeper節(jié)點

# node1節(jié)點
[root@node1 ~]# cd /data/bigdata/zookeeper/bin
[root@node1 conf]# zkServer.sh start

# node2節(jié)點
[root@node2 ~]# cd /data/bigdata/zookeeper/bin
[root@node2 conf]# zkServer.sh start

# node3節(jié)點
[root@node3 ~]# cd /data/bigdata/zookeeper/bin
[root@node3 conf]# zkServer.sh start

# 相應(yīng)進(jìn)程(其它類似)
[root@node1 ~]# jps
23993 QuorumPeerMain
24063 Jps
[root@node1 ~]#

(2) 集群啟動

由于zookeeper沒有提供同時啟動集群中所有節(jié)點的執(zhí)行腳本，在生產(chǎn)中逐個節(jié)點啟動稍微有些麻煩，自定義一個腳本用來啟動集群中所有節(jié)點，如下：

[root@node1 bigdata]# cat zookeeper_all_op.sh 
#!/bin/bash
# start zookeeper
zookeeperHome=/data/bigdata/src/zookeeper-3.4.12
zookeeperArr=( "node1" "node2" "node3" )
for znode in ${zookeeperArr[@]}; do
    ssh -p 22 -q root@$znode \
    "
        export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
        source /etc/profile
        $zookeeperHome/bin/zkServer.sh $1
    "
    echo "$znode zookeeper $1 done"
done

# 啟動
[root@node1 bigdata]#  ./zookeeper_all_op.sh start       

# 查看： leader為領(lǐng)導(dǎo)者（一臺）， follower為追隨者；
[root@node1 bigdata]# ./zookeeper_all_op.sh status
ZooKeeper JMX enabled by default
Using config: /data/bigdata/src/zookeeper-3.4.12/bin/../conf/zoo.cfg
Mode: follower
node1 zookeeper status done
ZooKeeper JMX enabled by default
Using config: /data/bigdata/src/zookeeper-3.4.12/bin/../conf/zoo.cfg
Mode: leader
node2 zookeeper status done
ZooKeeper JMX enabled by default
Using config: /data/bigdata/src/zookeeper-3.4.12/bin/../conf/zoo.cfg
Mode: follower
node3 zookeeper status done
[root@node1 bigdata]#

(3) 啟動客戶端腳本

[root@node1 bigdata]# ./zookeeper/bin/zkCli.sh  -server node2:2181
Connecting to node2:2181
2018-06-13 17:28:21,115 [myid:] - INFO  [main:Environment@100] - Client environment:zookeeper.version=3.4.12-e5259e437540f349646870ea94dc2658c4e44b3b, built on 03/27/2018 03:55 GMT
2018-06-13 17:28:21,119 [myid:] - INFO  [main:Environment@100] - Client environment:host.name=node1
......
省略
......
2018-06-13 17:28:21,220 [myid:] - INFO  [main-SendThread(node2:2181):ClientCnxn$SendThread@878] - Socket connection established to node2/192.168.1.102:2181, initiating session
2018-06-13 17:28:21,229 [myid:] - INFO  [main-SendThread(node2:2181):ClientCnxn$SendThread@1302] - Session establishment complete on server node2/192.168.1.102:2181, sessionid = 0x20034776a7c000e, negotiated timeout = 30000

WATCHER::

WatchedEvent state:SyncConnected type:None path:null
[zk: node2:2181(CONNECTED) 0] help
ZooKeeper -server host:port cmd args
    stat path [watch]
    set path data [version]
    ls path [watch]
    delquota [-n|-b] path
    ls2 path [watch]
    setAcl path acl
    setquota -n|-b val path
    history 
    redo cmdno
    printwatches on|off
    delete path [version]
    sync path
    listquota path
    rmr path
    get path [watch]
    create [-s] [-e] path data acl
    addauth scheme auth
    quit 
    getAcl path
    close 
    connect host:port
[zk: node2:2181(CONNECTED) 1] quit
Quitting...
2018-06-13 17:28:37,984 [myid:] - INFO  [main:ZooKeeper@687] - Session: 0x20034776a7c000e closed
2018-06-13 17:28:37,986 [myid:] - INFO  [main-EventThread:ClientCnxn$EventThread@521] - EventThread shut down for session: 0x20034776a7c000e
[root@node1 bigdata]#

2、啟動所有journalnode節(jié)點

# node1節(jié)點
[root@node1 ~]# cd /data/bigdata/hadoop/
[root@node1 hadoop]# ./sbin/hadoop-daemon.sh start journalnode
starting journalnode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-journalnode-node1.out
# 相應(yīng)進(jìn)程(其它類似)
[root@node1 ~]# jps
23993 QuorumPeerMain
24474 JournalNode                                  # 新啟動的進(jìn)程
24910 Jps
[root@node1 ~]#

# journalnode我配了3個
# node2節(jié)點
[root@node2 ~]# cd /data/bigdata/hadoop/
[root@node2 hadoop]# ./sbin/hadoop-daemon.sh start journalnode

# node3節(jié)點
[root@node3 ~]# cd /data/bigdata/hadoop/
[root@node3 hadoop]# ./sbin/hadoop-daemon.sh start journalnode

3、格式化namenode目錄(主節(jié)點node1)

[root@node1 hadoop]# cd /data/bigdata/hadoop
[root@node1 hadoop]# ./bin/hdfs namenode -format
    18/06/04 14:24:03 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = node1/192.168.1.101
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.7.6
..........................................
省略若干
...........................................
18/06/04 14:24:05 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
18/06/04 14:24:05 INFO util.ExitUtil: Exiting with status 0
18/06/04 14:24:05 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at node1/192.168.1.101
************************************************************/
[root@node1 hadoop]#

4、啟動當(dāng)前格式化的namenode進(jìn)程(主節(jié)點node1)

[root@node1 hadoop]# ./sbin/hadoop-daemon.sh start namenode
starting namenode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-namenode-node1.out
[root@node1 ~]# jps                      # # 相應(yīng)進(jìn)程
25155 Jps
23993 QuorumPeerMain
25050 NameNode                                 # name節(jié)點
24474 JournalNode
[root@node1 ~]#

5、在沒有格式化的NN上執(zhí)行同步命令(副節(jié)點node2)

[root@node2 hadoop]# ./bin/hdfs namenode -bootstrapStandby
    18/06/04 14:26:55 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = node2/192.168.1.102
STARTUP_MSG:   args = [-bootstrapStandby]
STARTUP_MSG:   version = 2.7.6
..........................................
省略若干
...........................................
************************************************************/
18/06/04 14:26:55 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
18/06/04 14:26:55 INFO namenode.NameNode: createNameNode [-bootstrapStandby]
18/06/04 14:26:55 WARN common.Util: Path /data/bigdata/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
18/06/04 14:26:55 WARN common.Util: Path /data/bigdata/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
=====================================================
About to bootstrap Standby ID nn2 from:
           Nameservice ID: mycluster
        Other Namenode ID: nn1
  Other NN's HTTP address: http://node1:50070
  Other NN's IPC  address: node1/192.168.1.101:8020
             Namespace ID: 736429223
            Block pool ID: BP-1022667957-192.168.1.101-1528093445721
               Cluster ID: CID-9d4854cd-7201-4e0d-9536-36e73195dc5a
           Layout version: -63
       isUpgradeFinalized: true
=====================================================
18/06/04 14:26:56 INFO common.Storage: Storage directory /data/bigdata/hdfs/name has been successfully formatted.
18/06/04 14:26:56 WARN common.Util: Path /data/bigdata/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
18/06/04 14:26:56 WARN common.Util: Path /data/bigdata/hdfs/name should be specified as a URI in configuration files. Please update hdfs configuration.
18/06/04 14:26:57 INFO namenode.TransferFsImage: Opening connection to http://node1:50070/imagetransfer?getimage=1&txid=0&storageInfo=-63:736429223:0:CID-9d4854cd-7201-4e0d-9536-36e73195dc5a
18/06/04 14:26:57 INFO namenode.TransferFsImage: Image Transfer timeout configured to 60000 milliseconds
18/06/04 14:26:57 INFO namenode.TransferFsImage: Transfer took 0.00s at 0.00 KB/s
18/06/04 14:26:57 INFO namenode.TransferFsImage: Downloaded file fsimage.ckpt_0000000000000000000 size 306 bytes.
18/06/04 14:26:57 INFO util.ExitUtil: Exiting with status 0
18/06/04 14:26:57 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at node2/192.168.1.102
************************************************************/
# 如果不成功，直接把node1節(jié)點的/data/bigdata/hdfs/name目錄復(fù)制過來，即可

# 啟動從節(jié)點
[root@node2 hadoop]# ./sbin/hadoop-daemon.sh start namenode
starting namenode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-namenode-node2.out

6、格式化ZKFC

格式化zkfc,讓在zookeeper中生成ha節(jié)點，在master上執(zhí)行如下命令，完成格式化：
[root@node1 hadoop]# ./bin/hdfs zkfc -formatZK
18/06/04 16:53:15 INFO tools.DFSZKFailoverController: Failover controller configured for NameNode NameNode at node1/192.168.1.101:8020
18/06/04 16:53:16 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT
18/06/04 16:53:16 INFO zookeeper.ZooKeeper: Client environment:host.name=node1
18/06/04 16:53:16 INFO zookeeper.ZooKeeper: Client environment:java.version=1.8.0_162
18/06/04 16:53:16 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
18/06/04 16:53:16 INFO zookeeper.ZooKeeper: Client environment:java.home=/opt/jdk1.8.0_162/jre
..........................................
省略若干
...........................................
18/06/04 16:53:16 INFO ha.ActiveStandbyElector: Session connected.
18/06/04 16:53:16 INFO ha.ActiveStandbyElector: Successfully created /hadoop-ha/mycluster in ZK.
18/06/04 16:53:16 INFO zookeeper.ZooKeeper: Session: 0x20034776a7c0000 closed
18/06/04 16:53:16 INFO zookeeper.ClientCnxn: EventThread shut down

[root@node1 hadoop]# ./sbin/hadoop-daemon.sh start zkfc
starting zkfc, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-zkfc-node1.out
[root@node1 hadoop]# jps
5443 DFSZKFailoverController                 # 新進(jìn)程
4664 JournalNode
23993 QuorumPeerMain
5545 Jps
4988 NameNode
[root@node1 hadoop]#

# 另一個節(jié)點啟動zkfc，有namenode運行的節(jié)點，都要啟動ZKFC
[root@node2 hadoop]# ./sbin/hadoop-daemon.sh start zkfc

7、啟動hdfs（datanode）

[root@node1 hadoop]# ./sbin/start-dfs.sh
Starting namenodes on [node1 node2]
node1: namenode running as process 25050. Stop it first.
node2: namenode running as process 30976. Stop it first.
node1: starting datanode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-datanode-node1.out
node2: starting datanode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-datanode-node2.out
node5: starting datanode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-datanode-node5.out
node3: starting datanode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-datanode-node3.out
node4: starting datanode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-datanode-node4.out
Starting journal nodes [node1 node2 node3]
node1: journalnode running as process 24474. Stop it first.
node3: journalnode running as process 19893. Stop it first.
node2: journalnode running as process 29871. Stop it first.
Starting ZK Failover Controllers on NN hosts [node1 node2]
node1: starting zkfc, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-zkfc-node1.out
node2: starting zkfc, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-zkfc-node2.out
[root@node1 hadoop]# jps
25968 DataNode                       # 所有節(jié)點都有
23993 QuorumPeerMain
5443 DFSZKFailoverController
25050 NameNode
24474 JournalNode
26525 Jps
[root@node1 hadoop]#

8、啟動yarn：

[root@node1 hadoop]# ./sbin/start-yarn.sh   
    starting yarn daemons
starting resourcemanager, logging to /data/bigdata/src/hadoop-2.7.6/logs/yarn-root-resourcemanager-node1.out
node1: starting nodemanager, logging to /data/bigdata/src/hadoop-2.7.6/logs/yarn-root-nodemanager-node1.out
node2: starting nodemanager, logging to /data/bigdata/src/hadoop-2.7.6/logs/yarn-root-nodemanager-node2.out
node4: starting nodemanager, logging to /data/bigdata/src/hadoop-2.7.6/logs/yarn-root-nodemanager-node4.out
node3: starting nodemanager, logging to /data/bigdata/src/hadoop-2.7.6/logs/yarn-root-nodemanager-node3.out
node5: starting nodemanager, logging to /data/bigdata/src/hadoop-2.7.6/logs/yarn-root-nodemanager-node5.out
[root@node1 hadoop]# jps
25968 DataNode
23993 QuorumPeerMain
5443 DFSZKFailoverController
25050 NameNode
24474 JournalNode
27068 Jps
26894 NodeManager            # 所有節(jié)點都有
[root@node1 hadoop]#

9、兩臺resourcemanager上啟動resourcemanager

（1）單獨啟動

[root@node3 hadoop]# ./sbin/yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /data/bigdata/src/hadoop-2.7.6/logs/yarn-root-resourcemanager-node3.out
[root@node4 hadoop]# ./sbin/yarn-daemon.sh start resourcemanager

# 相應(yīng)進(jìn)程(其它類似)
[root@node3 ~]# jps
21088 NodeManager
21297 ResourceManager           # 此進(jìn)程
19459 QuorumPeerMain
19893 JournalNode
20714 DataNode
21535 Jps
[root@node3 ~]#

（2）集群啟動

生產(chǎn)中一個hdfs集群會有兩個ResourceManager節(jié)點，若逐個節(jié)點啟動稍微有些麻煩，自定義一個腳本用來啟動集群中所有ResourceManager節(jié)點，如下：

[root@node1 bigdata]# pwd
/data/bigdata
[root@node1 bigdata]# cat yarn_all_resourcemanager.sh
#!/bin/bash
# resourcemanager management
hadoop_yarn_daemon_home=/data/bigdata/hadoop/sbin/
yarn_resourcemanager_node=( "node3" "node4" )
for renode in ${yarn_resourcemanager_node[@]}; do
    ssh -p 22 -q root@$znode "
        export PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
        source /etc/profile
        cd $hadoop_yarn_daemon_home/ && ./yarn-daemon.sh $1 resourcemanager
    "
    echo "$renode resourcemanager $1 done"
done
[root@node1 bigdata]# 

HDFS和yarn的web控制臺默認(rèn)監(jiān)聽端口分別為50070和8088?？梢酝ㄟ^瀏覽放訪問查看運行情況。
例：
        http://192.168.1.101:50070/
        http://192.168.1.103:8088/

# 停止命令：不操作
$HADOOP_HOME/sbin/stop-dfs.sh
$HADOOP_HOME/sbin/stop-yarn.sh

# 如果一切正常，使用jps可以查看到正在運行的Hadoop服務(wù)，機(jī)器上的顯示結(jié)果為：
[root@node1 hadoop]# jps
7312 Jps
1793 NameNode
2163 JournalNode
357 NodeManager
2696 QuorumPeerMain
14428 DFSZKFailoverController
1917 DataNode

到目前為止所啟動的進(jìn)程：

\	node1	node2	node3	node4	node5
JDK	?	?	?	?	?
QuorumPeerMain	?	?	?
JournalNode	?	?	?
NameNode	?	?
DFSZKFailoverController	?	?
DataNode	?	?	?	?	?
NodeManager	?	?	?	?	?
ResourceManager			?	?

10、驗證HDFS的HA功能

在任意一臺namenode機(jī)器上通過jps命令查找到namenode的進(jìn)程號，然后通過kill -9的方式殺掉進(jìn)程，觀察另一個namenode節(jié)點是否會從狀態(tài)standby變成active狀態(tài)。

[root@node1 bigdata]# jps
16704 JournalNode
16288 NameNode
16433 DataNode
23993 QuorumPeerMain
17241 NodeManager
18621 Jps
16942 DFSZKFailoverController
[root@node1 bigdata]# kill -9 16288

然后觀察原來是standby狀態(tài)的namenode機(jī)器的zkfc日志，若最后一行出現(xiàn)如下日志，則表示切換成功：

2018-05-31 16:14:41,114 INFOorg.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNodeat hd0/192.168.1.102:53310 to active state

這時再通過命令啟動被kill掉的namenode進(jìn)程

[root@node1 bigdata]# ./sbin/hadoop-daemon.sh start namenode

對應(yīng)進(jìn)程的zkfc最后一行日志如下：

2018-05-31 16:14:55,683 INFOorg.apache.hadoop.ha.ZKFailoverController: Successfully transitioned NameNodeat hd2/192.168.1.101:53310 to standby state

可以在兩臺namenode機(jī)器之間來回kill掉namenode進(jìn)程以檢查HDFS的HA配置！

七、scala

1、配置前準(zhǔn)備

scala運行在jvm虛擬機(jī)，需要配置jdk；

2、解壓

[root@node1 sbin]# tar -zxvf /data/tools/scala-2.11.12.tgz -C /data/bigdata/src
[root@node1 sbin]# ln -s /data/bigdata/src/scala-2.11.12 /data/bigdata/scala

# 添加環(huán)境變量
[root@node1 ~]# echo  -e "\n# scala\nexport scala_HOME=/data/bigdata/scala\nexport PATH=\$scala_HOME/bin:\$PATH" >> /etc/profile.d/bigdata_path.sh
[root@node1 ~]# cat /etc/profile.d/bigdata_path.sh
# zookeeper
export ZOOKEEPER_HOME=/data/bigdata/zookeeper
export PATH=$ZOOKEEPER_HOME/bin:$PATH

# hadoop
export HADOOP_HOME=/data/bigdata/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

# scala
export scala_HOME=/data/bigdata/scala
export PATH=$scala_HOME/bin:$PATH
[root@node1 ~]# source /etc/profile

3、查看scala版本

[root@node1 ~]#  scala -version
Scala code runner version 2.11.12 -- Copyright 2002-2016, LAMP/EPFL

4、運行scala命令

[root@node1 ~]# scala
Welcome to Scala 2.11.12 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_162).
Type in expressions for evaluation. Or try :help.

scala> 1+1
res0: Int = 2

scala>

如果以上兩步?jīng)]問題，表示scala已安裝和配置成功。

八、安裝spark

官方文檔

1、解壓spark

[root@node1 conf]# tar -zxvf /data/tools/spark-2.1.2-bin-hadoop2.7.tgz  -C /data/bigdata/src/
[root@node1 conf]# ln -s /data/bigdata/src/spark-2.1.2-bin-hadoop2.7  /data/bigdata/spark

# 添加環(huán)境變量
[root@node1 ~]# echo  -e "\n# spark\nexport SPARK_HOME=/data/bigdata/spark\nexport PATH=\$SPARK_HOME/bin:\$PATH" >> /etc/profile.d/bigdata_path.sh
[root@node1 ~]# cat /etc/profile.d/bigdata_path.sh
# zookeeper
export ZOOKEEPER_HOME=/data/bigdata/zookeeper
export PATH=$ZOOKEEPER_HOME/bin:$PATH

# hadoop
export HADOOP_HOME=/data/bigdata/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

# scala
export scala_HOME=/data/bigdata/scala
export PATH=$scala_HOME/bin:$PATH
[root@node1 ~]# source /etc/profile

# spark
export SPARK_HOME=/data/bigdata/spark
export PATH=$SPARK_HOME/bin:$PATH
[root@node1 ~]#

2、配置spark-env.sh

[root@node1 conf]# cd /data/bigdata/spark/conf/ 
[root@node1 conf]# cp spark-env.sh{.template,} 

# 添加：
[root@node1 conf]# vim spark-env.sh     
export SPARK_LOCAL_IP="192.168.1.101"           # 從節(jié)點改為自己的IP（或127.0.0.1 ），或者注掉
export SPARK_MASTER_IP="192.168.1.101"         
export JAVA_HOME=/opt/java
export SPARK_PID_DIR=/data/bigdata/hdfs/pids
export SPARK_LOCAL_DIRS= /data/bigdata/sparktmp
export PYSPARK_PYTHON=/usr/local/bin/python3    # 當(dāng)用python3開發(fā)時，配上python3的絕對路徑。

# 設(shè)置內(nèi)存，本節(jié)點可以調(diào)用的內(nèi)存
export SPARK_WORKER_MEMORY=58g
export SPARK_MASTER_PORT=7077
export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop
export LD_LIBRARY_PATH=$HADOOP_HOME/lib/native
export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplications=3 -Dspark.history.fs.logDirectory=hdfs://mycluster/directory"
# 限制程序申請資源最大核數(shù)，本節(jié)點可以調(diào)用的cpu核數(shù)
export SPARK_MASTER_OPTS="-Dspark.deploy.defaultCores=16"
export SPARK_SSH_OPTS="-p 22 -o StrictHostKeyChecking=no $SPARK_SSH_OPTS"

3、配置spark-defaults.conf

[root@node1 conf]# cp spark-defaults.conf{.template,}
[root@node1 conf]# vim spark-defaults.conf
#添加
spark.serializer                   org.apache.spark.serializer.KryoSerializer
spark.eventLog.enabled     true
spark.eventLog.dir              hdfs://mycluster/directory

# 使用Python3+開發(fā)時配置；
park.executorEnv.PYTHONHASHSEED=0

# 新建對應(yīng)目錄
[root@node1 conf]# mkdir -pv /data/bigdata/sparktmp
[root@node1 conf]# hdfs dfs -mkdir /directory

說明：spark.executorEnv.PYTHONHASHSEED=0配置：
如果你使用的是Python3+，并且在Spark集群上使用distinct()，reduceByKey()，和join()這幾個函數(shù)時，就會觸發(fā)下面的異常：
```
Exception: Randomness of hash of string should be disabled via PYTHONHASHSEED
```
python創(chuàng)建遍歷對象對象時會對每個對象進(jìn)行隨機(jī)哈希創(chuàng)建索引。然而在一個集群上，每個節(jié)點計算時對某一個變量創(chuàng)建的索引值不同，會導(dǎo)致數(shù)據(jù)索引沖突。因此需要設(shè)置PYTHONHASHSEED來固定隨機(jī)種子，保證索引一致。參見：
Spark集群配置（4）：其他填坑雜項

4、配置slaves

[root@node1 conf]# cp slaves{.template,}
[root@node1 conf]#  vim slaves
node1
node2
node3
node4
node5

5、分發(fā)

[root@node1 conf]# scp -rp /data/bigdata/src/spark-2.1.2-bin-hadoop2.7  node2:/data/bigdata/src/
[root@node1 conf]# scp -rp /data/bigdata/src/spark-2.1.2-bin-hadoop2.7  node3:/data/bigdata/src/
[root@node1 conf]# scp -rp /data/bigdata/src/spark-2.1.2-bin-hadoop2.7  node4:/data/bigdata/src/
[root@node1 conf]# scp -rp /data/bigdata/src/spark-2.1.2-bin-hadoop2.7  node5:/data/bigdata/src/

# 分發(fā)創(chuàng)建的目錄
[root@node1 bigdata]# scp -rp /data/bigdata/{hdfs,sparktmp,tmp} node2:/data/bigdata/
[root@node1 bigdata]# scp -rp /data/bigdata/{hdfs,sparktmp,tmp} node3:/data/bigdata/
[root@node1 bigdata]# scp -rp /data/bigdata/{hdfs,sparktmp,tmp} node4:/data/bigdata/
[root@node1 bigdata]# scp -rp /data/bigdata/{hdfs,sparktmp,tmp} node5:/data/bigdata/

# 創(chuàng)建軟連接
[root@node2 ~]# ln -s /data/bigdata/src/spark-2.1.2-bin-hadoop2.7  /data/bigdata/spark
[root@node3 ~]# ln -s /data/bigdata/src/spark-2.1.2-bin-hadoop2.7  /data/bigdata/spark
[root@node4 ~]# ln -s /data/bigdata/src/spark-2.1.2-bin-hadoop2.7  /data/bigdata/spark
[root@node5 ~]# ln -s /data/bigdata/src/spark-2.1.2-bin-hadoop2.7  /data/bigdata/spark

# 修改$SPARK_HOME/conf/spark-env.sh中的SPARK_LOCAL_IP參數(shù)
[root@node2 ~]# vim $SPARK_HOME/conf/spark-env.sh
export SPARK_LOCAL_IP="192.168.1.102" 
[root@node3 ~]# vim $SPARK_HOME/conf/spark-env.sh
export SPARK_LOCAL_IP="192.168.1.103" 
[root@node4 ~]# vim $SPARK_HOME/conf/spark-env.sh
export SPARK_LOCAL_IP="192.168.1.104"   
[root@node5 ~]# vim $SPARK_HOME/conf/spark-env.sh
export SPARK_LOCAL_IP="192.168.1.105"

6、啟動spark

[root@node1 bin]# cd /data/bigdata/spark/sbin
[root@node1 sbin]# ./start-all.sh
starting org.apache.spark.deploy.master.Master, logging to /data/bigdata/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-node1.out
node1: starting org.apache.spark.deploy.worker.Worker, logging to /data/bigdata/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-node1.out
node5: starting org.apache.spark.deploy.worker.Worker, logging to /data/bigdata/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-node5.out
node3: starting org.apache.spark.deploy.worker.Worker, logging to /data/bigdata/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-node3.out
node4: starting org.apache.spark.deploy.worker.Worker, logging to /data/bigdata/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-node4.out
node2: starting org.apache.spark.deploy.worker.Worker, logging to /data/bigdata/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-node2.out
[root@node1 sbin]# jps
5443 DFSZKFailoverController
5684 Master                   # spark的master
1092 HRegionServer
5846 Worker                   # spark進(jìn)程
904 HMaster
4664 JournalNode
23993 QuorumPeerMain
6266 Jps
7227 NodeManager
4988 NameNode
6495 DataNode

web界面：http://192.168.1.101:8080/

[root@node1 sbin]# ./start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to /data/bigdata/spark/logs/spark-root-org.apache.spark.deploy.history.HistoryServer-1-node1.out
[root@node1 sbin]# jps
5443 DFSZKFailoverController
5684 Master                   # spark的master
1092 HRegionServer
5846 Worker                   # spark進(jìn)程
904 HMaster
4664 JournalNode
29366 HistoryServer         # spark 保存歷史日志記錄的
23993 QuorumPeerMain
6266 Jps
7227 NodeManager
4988 NameNode
6495 DataNode

訪問WEBUI： http://192.168.1.101:18080/

到目前為止所啟動的進(jìn)程：

\	node1	node2	node3	node4	node5
JDK	?	?	?	?	?
QuorumPeerMain	?	?	?
JournalNode	?	?	?
NameNode	?	?
DFSZKFailoverController	?	?
DataNode	?	?	?	?	?
NodeManager	?	?	?	?	?
ResourceManager			?	?
Master	?
Worker	?	?	?	?	?
HistoryServer	?

九、安裝hbase

Master和Hadoop的NameNode進(jìn)程運行在同一臺主機(jī)上，與DataNode通信
以讀寫HDFS的數(shù)據(jù)。RegionServer跟Hadoop的DataNode運行在同一臺主機(jī)上。

參考：
官方文檔
hbase 數(shù)據(jù)庫簡介安裝與常用命令的使用

1、解壓hbase

[root@node1 sbin]# tar -zxvf  /data/tools/hbase-1.2.6-bin.tar.gz  -C /data/bigdata/src
[root@node1 sbin]# ln -s  /data/bigdata/src/hbase-1.2.6  /data/bigdata/hbase

# 添加環(huán)境變量
[root@node1 ~]# echo  -e "\n# hbase\nexport HBASE_HOME=/data/bigdata/hbase\nexport PATH=\$HBASE_HOME/bin:\$PATH" >> /etc/profile.d/bigdata_path.sh
[root@node1 ~]# cat /etc/profile.d/bigdata_path.sh
# zookeeper
export ZOOKEEPER_HOME=/data/bigdata/zookeeper
export PATH=$ZOOKEEPER_HOME/bin:$PATH

# hadoop
export HADOOP_HOME=/data/bigdata/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

# scala
export scala_HOME=/data/bigdata/scala
export PATH=$scala_HOME/bin:$PATH

# spark
export SPARK_HOME=/data/bigdata/spark
export PATH=$SPARK_HOME/bin:$PATH

# hbase
export HBASE_HOME=/data/bigdata/hbase
export PATH=$HBASE_HOME/bin:$PATH
[root@node1 ~]#

2、修改$HBASE_HOME/conf/hbase-env.sh,添加

[root@node1 sbin]#  cd /data/bigdata/hbase/conf
[root@node1 conf]#  vim hbase-env.sh
export JAVA_HOME=/opt/java
export HBASE_HOME=/data/bigdata/hbase
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$HADOOP_HOME/lib/native/
export HBASE_LIBRARY_PATH=$HBASE_LIBRARY_PATH:$HBASE_HOME/lib/native/
# 設(shè)置到Hadoop的etc/hadoop目錄是用來引導(dǎo)Hbase找到Hadoop,也就是說hbase和hadoop進(jìn)行關(guān)聯(lián)【必須設(shè)置,否則hmaster起不來】
export HBASE_CLASSPATH=$HADOOP_HOME/etc/hadoop
export HBASE_MANAGES_ZK=false           #不啟用hbase自帶的zookeeper   
export HBASE_PID_DIR=/data/bigdata/hdfs/pids
export HBASE_SSH_OPTS="-o ConnectTimeout=1 -p 22"           # ssh端口；

# jdk1.8及以上版本注掉下面兩行
# Configure PermSize. Only needed in JDK7. You can safely remove it for JDK8+
#export HBASE_MASTER_OPTS="$HBASE_MASTER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m"
#export HBASE_REGIONSERVER_OPTS="$HBASE_REGIONSERVER_OPTS -XX:PermSize=128m -XX:MaxPermSize=128m"

3、修改regionservers文件

[root@node1 conf]# vim regionservers
node1
node2
node3

4、修改hbase-site.xml文件

[root@node1 conf]# vim hbase-site.xml
<configuration>
    <property>
          <name>hbase.rootdir</name>
        <value>hdfs://mycluster/hbase</value>
    </property>
    <property>
           <name>hbase.zookeeper.quorum</name>
        <value>node1,node2,node3</value>
           <description>指定集群zookeeper主機(jī)名</description>
    </property> 
    <property>
            <name>hbase.zookeeper.property.clientPort</name>
            <value>2181</value>
     </property>
  <property>
            <name>hbase.master.info.port</name>
            <value>60010</value>
     </property>
   <property>
           <name>hbase.cluster.distributed</name>
           <value>true</value>
            <description>多臺hbase開啟此參數(shù)</description>
      </property>
</configuration>

5、分發(fā)

[root@node1 conf]# scp -rp /data/bigdata/src/hbase-1.2.6  node2:/data/bigdata/src/
[root@node1 conf]# scp -rp /data/bigdata/src/hbase-1.2.6  node3:/data/bigdata/src/

# 創(chuàng)建軟連接
[root@node2 ~]# ln -s  /data/bigdata/src/hbase-1.2.6  /data/bigdata/hbase
[root@node3 ~]# ln -s  /data/bigdata/src/hbase-1.2.6  /data/bigdata/hbase

6、啟動hbase

[root@node1 hadoop]# cd /data/bigdata/hbase/bin
[root@node1 bin]# ./start-hbase.sh
[root@node1 bin]# jps
5443 DFSZKFailoverController
1092 HRegionServer      # hbase進(jìn)程
904 HMaster                   # hbase主節(jié)點
4664 JournalNode
23993 QuorumPeerMain
14730 Master
7227 NodeManager
4988 NameNode
14877 Worker
1917 Jps
6495 DataNode
[root@node1 bin]#

到目前為止所啟動的進(jìn)程：

\	node1	node2	node3	node4	node5
JDK	?	?	?	?	?
QuorumPeerMain	?	?	?
JournalNode	?	?	?
NameNode	?	?
DFSZKFailoverController	?	?
DataNode	?	?	?	?	?
NodeManager	?	?	?	?	?
ResourceManager			?	?
Master	?
Worker	?	?	?	?	?
HistoryServer	?
HMaster	?
HRegionServer	?	?	?

Hbase web頁面http://192.168.1.101:16030
Hbase Master URL:http://192.168.1.101:60010

# 測試
[root@node1 bin]# ./hbase shell
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data/bigdata/src/hbase-1.2.6/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data/bigdata/src/hadoop-2.7.6/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
HBase Shell; enter 'help<RETURN>' for list of supported commands.
Type "exit<RETURN>" to leave the HBase Shell
Version 1.2.6, rUnknown, Mon May 29 02:25:32 CDT 2017

hbase(main):001:0> list                         # 輸入命令list
TABLE                                                                                                                                                                                          
0 row(s) in 0.2670 seconds

=> []
hbase(main):002:0> quit
[root@node1 bin]#

十、kafka

官方文檔
kafka實戰(zhàn)最佳經(jīng)驗

1、解壓創(chuàng)建環(huán)境變量

[root@node4 conf]# tar -zxvf /data/tools/kafka_2.11-1.1.0.tgz  -C /data/bigdata/src/
[root@node4 conf]# ln -s /data/bigdata/src/kafka_2.11-1.1.0  /data/bigdata/kafka

# 添加環(huán)境變量
[root@node4 ~]# echo  -e "\n# kafka\nexport KAFKA_HOME=/data/bigdata/kafka\nexport PATH=\$KAFKA_HOME/bin:\$PATH" >> /etc/profile.d/bigdata_path.sh
[root@node4 ~]# cat /etc/profile.d/bigdata_path.sh
# zookeeper
export ZOOKEEPER_HOME=/data/bigdata/zookeeper
export PATH=$ZOOKEEPER_HOME/bin:$PATH

# hadoop
export HADOOP_HOME=/data/bigdata/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

# hbase
export HBASE_HOME=/data/bigdata/hbase
export PATH=$HBASE_HOME/bin:$PATH

# scala
export scala_HOME=/data/bigdata/scala
export PATH=$scala_HOME/bin:$PATH

# spark
export SPARK_HOME=/data/bigdata/spark
export PATH=$SPARK_HOME/bin:$PATH

# kafka
export KAFKA_HOME=/data/bigdata/kafka
export PATH=$KAFKA_HOME/bin:$PATH

# 生效
[root@node4 conf]# source /etc/profile

2、修改server.properties配置文件：

[root@node4 ~]# cd /data/bigdata/kafka/config/                       # 進(jìn)入conf目錄
[root@node4 conf]# cp  server.properties{,.bak}                      # 備份配置文件
[root@node4 conf]# vim server.properties
broker.id=1
listeners=PLAINTEXT://node4:9092
advertised.listeners=PLAINTEXT://node4:9092
log.dirs=/data/bigdata/kafka/logs
zookeeper.connect=node1:2181,node2:2181,node3:2181

[root@node4 conf]# mkdir -p /data/bigdata/kafka/logs

3、分發(fā)

[root@node4 conf]# scp -rp /data/bigdata/src/kafka_2.11-1.1.0 node5: /data/bigdata/src/

# 子節(jié)點
[root@node5 ~]# ln -s /data/bigdata/src/kafka_2.11-1.1.0  /data/bigdata/kafka
[root@node5 ~]# mkdir -p /data/bigdata/kafka/logs

# 修改每個server.properties文件中的broker.id
[root@node5 ~]# cd /data/bigdata/kafka/
[root@node5 kafka]# vim ./config/server.properties
broker.id=2
listeners=PLAINTEXT://node5:9092
advertised.listeners=PLAINTEXT://node5:9092

# 查看
[root@node1 conf]# ansible kafka -m shell -a '$(which egrep) --color=auto "^broker.id|^listeners|^advertised.listeners" ${KAFKA_HOME}/config/server.properties'

4、啟動

[root@node4 kafka]# ./bin/kafka-server-start.sh config/server.properties                     # 單個節(jié)點前臺運行
[2018-06-05 13:32:35,323] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2018-06-05 13:32:35,672] INFO starting (kafka.server.KafkaServer)
[2018-06-05 13:32:35,673] INFO Connecting to zookeeper on node1:2181,node2:2181,node3:2181 (kafka.server.KafkaServer)
[2018-06-05 13:32:35,692] INFO [ZooKeeperClient] Initializing a new session to node1:2181,node2:2181,node3:2181. (kafka.zookeeper.ZooKeeperClient)
[2018-06-05 13:32:35,698] INFO Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT 
..........................................
省略若干
...........................................
[2018-06-05 13:33:38,920] INFO Terminating process due to signal SIGINT (kafka.Kafka$)
[2018-06-05 13:33:39,168] INFO [ThrottledRequestReaper-Produce]: Stopped (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-06-05 13:33:39,168] INFO [ThrottledRequestReaper-Produce]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-06-05 13:33:39,168] INFO [ThrottledRequestReaper-Request]: Shutting down (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-06-05 13:33:39,168] INFO [ThrottledRequestReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-06-05 13:33:39,168] INFO [ThrottledRequestReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2018-06-05 13:33:39,169] INFO [SocketServer brokerId=1] Shutting down socket server (kafka.network.SocketServer)
[2018-06-05 13:33:39,193] INFO [SocketServer brokerId=1] Shutdown completed (kafka.network.SocketServer)
[2018-06-05 13:33:39,199] INFO [KafkaServer id=1] shut down completed (kafka.server.KafkaServer)

[root@node4 kafka]# ./bin/kafka-server-start.sh -daemon config/server.properties             # 后臺啟動單個節(jié)點，其它kafka也要啟動，進(jìn)程名為：Kafka

到目前為止所啟動的進(jìn)程：

\	node1	node2	node3	node4	node5
JDK	?	?	?	?	?
QuorumPeerMain	?	?	?
JournalNode	?	?	?
NameNode	?	?
DFSZKFailoverController	?	?
DataNode	?	?	?	?	?
NodeManager	?	?	?	?	?
ResourceManager			?	?
HMaster	?
HRegionServer	?	?	?
HistoryServer	?
Master	?
Worker	?	?	?	?	?
Kafka				?	?

Kafka并沒有提供同時啟動集群中所有節(jié)點的執(zhí)行腳本，在生產(chǎn)中一個Kafka集群往往會有多個節(jié)點，若逐個節(jié)點啟動稍微有些麻煩，自定義一個腳本用來啟動集群中所有節(jié)點，如下：

[root@node1 bigdata]# cat kafka_cluster_start.sh 
#!/bin/bash

brokers="node4 node5"
KAFKA_HOME="/data/bigdata/kafka"

for broker in $brokers
  do
    ssh $broker -C "source /etc/profile; cd ${KAFKA_HOME}/bin && ./kafka-server-start.sh -daemon ../config/server.properties"

    if [ $? -eq 0 ]; then
       echo "INFO:[${broker}] Start successfully "
    fi
done
[root@node1 bigdata]#

5、測試

創(chuàng)建主題：（指明要連接的zookeeper）例如主題名稱為：TestTopic

[root@node4 kafka]# ./bin/kafka-topics.sh --create --zookeeper node1:2181,node2:2181,node3:2181 --replication-factor 1 --partitions 1 --topic TestTopic
Created topic "TestTopic".                       # 表示創(chuàng)建成功

查看主題：

[root@node4 kafka]# ./bin/kafka-topics.sh --list --zookeeper node1:2181,node2:2181,node3:2181
TestTopic     # 可以看到所有已創(chuàng)建的主題

任選一臺，創(chuàng)建生產(chǎn)者：（kafka集群用戶）

[root@node4 kafka]# ./bin/kafka-console-producer.sh --broker-list node4:9092,node5:9092 --topic TestTopic
>hi
>hello 1
>hello 2
>

另一臺，創(chuàng)建消費者

[root@node5 kafka]# ./bin/kafka-console-consumer.sh --bootstrap-server  node4:9092,node5:9092 --from-beginning --topic TestTopic
hi
hello 1
hello 2

生產(chǎn)者輸入一些數(shù)據(jù)，看消費者是否顯示生產(chǎn)者所輸入的數(shù)據(jù)。

6、關(guān)閉

[root@node4 kafka]# ./bin/kafka-server-stop.sh             # 關(guān)閉，其它kafka也要關(guān)閉，
有時候不管用，顯示“No kafka server to stop”，
失敗的原因是kafka-server-stop.sh腳本里的ps ax | grep -i 'kafka.Kafka' | grep Java | grep -v grep | awk '{print $1}'命令在我所使用的操作系統(tǒng)中并不能得到Kafka進(jìn)程的PID：
因此這里將kafka-server-stop.sh腳本查找PID的命令修改如下：

#PIDS=$(ps ax | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk '{print $1}')     # 注掉，改為下面命令
PIDS=$(jps | grep -i 'Kafka' |awk '{print $1}')

Kafka也同樣沒有提供關(guān)閉集群操作的腳本。這里我提供一個用來關(guān)閉Kafka集群的腳本（可以放在任意一條節(jié)點上）：

[root@node1 bigdata]# cat kafka_cluster_stop.sh 
#!/bin/bash

brokers="node4 node5"
KAFKA_HOME="/data/bigdata/kafka"

for broker in $brokers
  do
    ssh $broker -C "cd ${KAFKA_HOME}/bin && ./kafka-server-stop.sh"

    if [ $? -eq 0 ]; then
       echo "INFO:[${broker}] shut down completed "
    fi

done
[root@node1 bigdata]#  chmod +x kafka-cluster-stop.sh

十一、hive

官方文檔

1、安裝mysql數(shù)據(jù)庫

參考：https://blog.51cto.com/moerjinrong/2092614

# 新建hive用戶及metastore庫
root@node2 14:37:  [(none)]> grant all privileges on *.* to 'hive'@'192.168.1.%' identified by '123456';
Query OK, 0 rows affected, 1 warning (0.00 sec)

root@node2 15:14:  [(none)]> create database metastore;     # 待定
Query OK, 1 row affected (0.01 sec)

2、解壓添加環(huán)境變量

[root@node2 ~]# tar -zxvf /data/tools/apache-hive-2.3.3-bin.tar.gz  -C /data/bigdata/src/
[root@node2 ~]# ln -s /data/bigdata/src/apache-hive-2.3.3-bin  /data/bigdata/hive

# 添加環(huán)境變量
[root@node2 ~]# echo  -e "\n# hive\nexport HIVE_HOME=/data/bigdata/hive\nexport PATH=\$HIVE_HOME/bin:\$PATH" >> /etc/profile.d/bigdata_path.sh
[root@node2 ~]# cat /etc/profile.d/bigdata_path.sh
# zookeeper
export ZOOKEEPER_HOME=/data/bigdata/zookeeper
export PATH=$ZOOKEEPER_HOME/bin:$PATH

# hadoop
export HADOOP_HOME=/data/bigdata/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

# hbase
export HBASE_HOME=/data/bigdata/hbase
export PATH=$HBASE_HOME/bin:$PATH

# scala
export scala_HOME=/data/bigdata/scala
export PATH=$scala_HOME/bin:$PATH

# spark
export SPARK_HOME=/data/bigdata/spark
export PATH=$SPARK_HOME/bin:$PATH

# kafka
export KAFKA_HOME=/data/bigdata/kafka
export PATH=$KAFKA_HOME/bin:$PATH

# hive
export HIVE_HOME=/data/bigdata/hive
export PATH=$HIVE_HOME/bin:$PATH"

# 生效
[root@node2 ~]# source /etc/profile

3、hdfs上新建目錄

在hdfs中新建目錄/user/hive/warehouse
首先啟動hadoop任務(wù)
hdfs dfs -mkdir /tmp
hdfs dfs -mkdir /user
hdfs dfs -mkdir /user/hive
hdfs dfs -mkdir /user/hive/warehouse

hadoop fs -chmod g+w /tmp
hadoop fs -chmod g+w /user/hive/warehouse

# 將mysql的驅(qū)動jar包mysql-connector-java-5.1.46.jar拷入hive的lib目錄下面，沒有就下載：
wget -P $HIVE_HOME/lib http://central.maven.org/maven2/mysql/mysql-connector-java/5.1.46/mysql-connector-java-5.1.46.jar

4、修改server.properties配置文件：

[root@node2 ~]# cd  /data/bigdata/hive/conf/
[root@node2 conf]#  cp hive-default.xml.template hive-site.xml
[root@node2 conf]#  vim hive-site.xml
# 修改下列屬性值（通過/指令尋找，如果第一個定位不正確，n尋找下一個）
<property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://192.168.1.102:3306/hive?createDatabaseIfNotExist=true&useSSL=false</value>     # mysql沒有開啟ssl
    <description>
                JDBC connect string for a JDBC metastore;
                '?'符號是在URL后通過get方法傳遞參數(shù)的起始標(biāo)志，
                多個參數(shù)之間可用'&'符號連接，因為這些字符對于HTML有特殊意義，
                所以在Java中要用到轉(zhuǎn)義字符使用它，而&在HTML中就會被轉(zhuǎn)義為'&'符號，用于參數(shù)連接。
    </description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
    <description>Username to use against metastore database</description>
  </property>
  <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>123456</value>
    <description>password to use against metastore database</description>
  </property>

    <property>
   <name>hive.metastore.warehouse.dir</name>
   <value>/data/bigdata/hive/warehouse</value>
    </property>
    <property>
        <name>hive.metastore.local</name>
        <value>true</value>
    </property>
  <property>
    <name>hive.exec.local.scratchdir</name>
    <value>/data/bigdata/hive/tmp</value>
    <description>Local scratch space for Hive jobs</description>
  </property>
  <property>
    <name>hive.downloaded.resources.dir</name>
    <value>/data/bigdata/hive/tmp/resources</value>
    <description>Temporary local directory for added resources in the remote file system.</description>
  </property>
  <property>
    <name>hive.querylog.location</name>
    <value>/data/bigdata/hive/tmp</value>
    <description>Location of Hive run time structured log file</description>
  </property>
  <property>
    <name>hive.server2.logging.operation.log.location</name>
    <value>/data/bigdata/hive/tmp/operation_logs</value>
    <description>Top level directory where operation logs are stored if logging functionality is enabled</description>
  </property>

注意：由于HTML格式問題，上面jdbc的URL中的&改為照片中紅色下劃線的符號
大數(shù)據(jù)Hadoop集群搭建

# 創(chuàng)建指定目錄
[root@node2 conf]#  mkdir -pv /data/bigdata/hive/{tmp/{operation_logs,resources},warehouse}

5、初始化并運行

# 使用schematool 初始化metastore的schema：
[root@node2 conf]#  schematool -initSchema -dbType mysql 
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/data/bigdata/src/apache-hive-2.3.3-bin/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/data/bigdata/src/hadoop-2.7.6/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL:    jdbc:mysql://127.0.0.1:3306/metastore?useSSL=false
Metastore Connection Driver :    com.mysql.jdbc.Driver
Metastore connection User:   hive
Starting metastore schema initialization to 2.3.0
Initialization script hive-schema-2.3.0.mysql.sql
Initialization script completed
schemaTool completed
[root@node2 conf]# 

# 運行hive
[root@node2 conf]#  hive                      # 對應(yīng)RunJar進(jìn)程
hive> show databases;
OK
default
Time taken: 1.881 seconds, Fetched: 1 row(s)
hive> use default;
OK
Time taken: 0.081 seconds
hive> create table kylin_test(test_count int);
OK
Time taken: 2.9 seconds
hive> show tables;
OK
Time taken: 0.151 seconds, Fetched: 1 row(s)
hive> quit;

到目前為止所啟動的進(jìn)程：

\	node1	node2	node3	node4	node5
JDK	?	?	?	?	?
QuorumPeerMain	?	?	?
JournalNode	?	?	?
NameNode	?	?
DFSZKFailoverController	?	?
DataNode	?	?	?	?	?
NodeManager	?	?	?	?	?
ResourceManager			?	?
HMaster	?
HRegionServer	?	?	?
HistoryServer	?
Master	?
Worker	?	?	?	?	?
Kafka				?	?
RunJar		?				# 啟動hive時才有

十二、kylin

官方文檔

1、安裝前準(zhǔn)備

安裝kylin前確保：hadoop 2.4+、hbase 0.13+、hive 0.98+，1.*已經(jīng)安裝并啟動。
Hive需要啟動metastore和hiveserver2。

Apache Kylin同樣可以使用集群部署，但使用集群部署并不能增加計算速度
因為計算過程使用MapReduce引擎，與Kylin自身無關(guān)，而是主要為查詢提供負(fù)載均衡。本次采用單節(jié)點。

2、解壓并創(chuàng)建環(huán)境變量

[root@node2 ~]# tar zxvf /data/tools/apache-kylin-2.3.1-hbase1x-bin.tar.gz  -C /data/bigdata/src/
[root@node2 ~]# ln -s /data/bigdata/src/apache-kylin-2.3.1-bin/  /data/bigdata/kylin

# 添加環(huán)境變量
[root@node2 ~]# echo  -e "\n# kylin\nexport KYLIN_HOME=/data/bigdata/kylin\nexport PATH=\$KYLIN_HOME/bin:\$PATH" >> /etc/profile.d/bigdata_path.sh
[root@node2 ~]# cat /etc/profile.d/bigdata_path.sh
# zookeeper
export ZOOKEEPER_HOME=/data/bigdata/zookeeper
export PATH=$ZOOKEEPER_HOME/bin:$PATH

# hadoop
export HADOOP_HOME=/data/bigdata/hadoop
export PATH=$HADOOP_HOME/bin:$HADOOP_HOME/sbin:$PATH

# hbase
export HBASE_HOME=/data/bigdata/hbase
export PATH=$HBASE_HOME/bin:$PATH

# scala
export scala_HOME=/data/bigdata/scala
export PATH=$scala_HOME/bin:$PATH

# spark
export SPARK_HOME=/data/bigdata/spark
export PATH=$SPARK_HOME/bin:$PATH

# kafka
export KAFKA_HOME=/data/bigdata/kafka
export PATH=$KAFKA_HOME/bin:$PATH

# kylin
export KYLIN_HOME=/data/bigdata/kylin
export PATH=$KYLIN_HOME/bin:$PATH"

# 生效
[root@node2 ~]# source /etc/profile

3、復(fù)制hive的相關(guān)jar到kylin

將hive安裝目錄lib目錄中的所有jar包復(fù)制到kylin安裝目錄下的lib目錄中。

[root@node2 ~]#  cp -a /data/bigdata/hive/lib/* /data/bigdata/kylin/lib/

4、配置Kylin使用的Hive數(shù)據(jù)庫：

[root@node2 ~]# cd /data/bigdata/kylin/conf
[root@node2 conf]# vim kylin.properties
kylin.server.cluster-servers=node2:7070                    # kylin集群設(shè)置，修改主機(jī)名或ip，端口
kylin.job.jar=$KYLIN_HOME/lib/kylin-job-2.3.1.jar     # 修改jar包版本及路徑
kylin.coprocessor.local.jar=$KYLIN_HOME/lib/kylin-coprocessor-2.3.1.jar         # 修改jar包版本及路徑

# List of web servers in use, this enables one web server instance to sync up with other servers
kylin.rest.servers=node2:7070

## 配置Kylin使用的Hive數(shù)據(jù)庫，這里配置在Hive中使用的schema，改為當(dāng)前用戶
kylin.job.hive.database.for.intermediatetable=root

5、如果沒有啟動https，請關(guān)閉

[root@node2 ~]# cd /data/bigdata/kylin/tomcat/conf
[root@node2 conf]# cp -a server.xml{,_$(date +%F)}
[root@node2 conf]# vim server.xml
 85                    maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
改為：
85                    maxThreads="150" SSLEnabled="false" scheme="https" secure="false"

如果不關(guān)閉，會報如下錯誤

SEVERE: Failed to load keystore type JKS with path conf/.keystore due to /data/bigdata/kylin/tomcat/conf/.keystore (No such file or directory)
java.io.FileNotFoundException: /data/bigdata/kylin/tomcat/conf/.keystore (No such file or directory)

6、修改$KYLIN_HOME/bin/kylin.sh

[root@node2 conf]# vim ../bin/kylin.sh
export KYLIN_HOME=/data/bigdata/kylin
export CATALINA_HOME=/data/bigdata/kylin/tomcat
export PATH=$CATALINA_HOME/bin:$PATH

export HCAT_HOME=$HIVE_HOME/hcatalog
export hive_dependency=$HIVE_HOME/conf:$HIVE_HOME/lib/*:$HCAT_HOME/share/hcatalog/hive-hcatalog-core-2.3.3.jar
export HBASE_CLASSPATH_PREFIX=$CATALINA_HOME/bin/bootstrap.jar:$CATALINA_HOME/bin/tomcatjuli.jar:$CATALINA_HOME/lib/*:$hive_dependency:$HBASE_CLASSPATH_PREFIX

#使用HDFS超級用戶在HDFS上為Kylin創(chuàng)建工作目錄，并賦權(quán)給服務(wù)器登錄名
#[root@node2 conf]# hdfs dfs -mkdir /kylin
#[root@node2 conf]# hdfs dfs -chown -R root:root /kylin

7、檢查kylin依賴

進(jìn)入bin目錄下分別執(zhí)行

[root@node2 bin]# cd $KYLIN_HOME/bin
[root@node2 bin]# ./check-env.sh
Retrieving hadoop conf dir...
KYLIN_HOME is set to /data/bigdata/kylin

[root@node2 bin]# ./find-hive-dependency.sh
Retrieving hive dependency...

[root@node2 bin]# ./find-hbase-dependency.sh
Retrieving hbase dependency...

8、啟動kylin服務(wù)

在kylin安裝根目錄下執(zhí)行

[root@node2 bin]# ./kylin.sh start
Retrieving hadoop conf dir...
KYLIN_HOME is set to /data/bigdata/kylin
Retrieving hive dependency...
Retrieving hbase dependency...
Retrieving hadoop conf dir...
Retrieving kafka dependency...
Retrieving Spark dependency...
Start to check whether we need to migrate acl tables
..........................................
省略若干
...........................................
2018-06-05 17:12:10,111 INFO  [Thread-6] zookeeper.ZooKeeper:684 : Session: 0x300346e6b9e000d closed
2018-06-05 17:12:10,111 INFO  [main-EventThread] zookeeper.ClientCnxn:512 : EventThread shut down
2018-06-05 17:12:10,210 INFO  [close-hbase-conn] client.ConnectionManager$HConnectionImplementation:2068 : Closing master protocol: MasterService
2018-06-05 17:12:10,211 INFO  [close-hbase-conn] client.ConnectionManager$HConnectionImplementation:1676 : Closing zookeeper sessionid=0x20034776a7c0004
2018-06-05 17:12:10,214 INFO  [close-hbase-conn] zookeeper.ZooKeeper:684 : Session: 0x20034776a7c0004 closed
2018-06-05 17:12:10,214 INFO  [main-EventThread] zookeeper.ClientCnxn:512 : EventThread shut down

A new Kylin instance is started by root. To stop it, run 'kylin.sh stop'
Check the log at /data/bigdata/kylin/logs/kylin.log
Web UI is at http://<hostname>:7070/kylin

[root@node2 bin]#

到目前為止所啟動的進(jìn)程：	\	node1	node2	node3	node4	node5
JDK	?	?	?	?	?
QuorumPeerMain	?	?	?
JournalNode	?	?	?
NameNode	?	?
DFSZKFailoverController	?	?
DataNode	?	?	?	?	?
NodeManager	?	?	?	?	?
ResourceManager			?	?
HMaster	?
HRegionServer	?	?	?
HistoryServer	?
Master	?
Worker	?	?	?	?	?
Kafka				?	?
RunJar		?				# 啟動hive時才有
RunJar		?	?			# kylin進(jìn)程

服務(wù)啟動后，瀏覽器訪問地址：http://IP:7070/kylin/
用戶名：ADMIN
密碼：KYLIN

9、配置hive數(shù)據(jù)源

1.配置數(shù)據(jù)源
（1）依次選擇 Model -> Data Source -> Load Hive Table
大數(shù)據(jù)Hadoop集群搭建

（2）輸入 hive 中數(shù)據(jù)庫的表名格式為：數(shù)據(jù)庫名.數(shù)據(jù)表名
如：db_hiveTest.student ，然后點擊Sync即可。
大數(shù)據(jù)Hadoop集群搭建

添加成功后，效果如下圖：
大數(shù)據(jù)Hadoop集群搭建

10、常見錯誤

1、界面無法同步hive表元數(shù)據(jù)
解決方法，在kylin安裝目錄下：
執(zhí)行命令：vim ./bin/kylin.sh 需要對此腳本做以下修改：

export HBASE_CLASSPATH_PREFIX=${tomcat_root}/bin/bootstrap.jar:${tomcat_root}/bin/tomcat-juli.jar:${tomcat_root}/lib/*:$hive_dependency:$HBASE_CLASSPATH_PREFIX# 在路徑中添加$hive_dependency。

十三、記一次日常啟動

1、zookeeper:

[root@node1 ~]# cd /data/bigdata/
[root@node1 bigdata]# ./zookeeper_all_op.sh start
ZooKeeper JMX enabled by default
Using config: /data/bigdata/src/zookeeper-3.4.12/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
node1 zookeeper start done
ZooKeeper JMX enabled by default
Using config: /data/bigdata/src/zookeeper-3.4.12/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
node2 zookeeper start done
ZooKeeper JMX enabled by default
Using config: /data/bigdata/src/zookeeper-3.4.12/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
node3 zookeeper start done

[root@node1 bigdata]# ./zookeeper_all_op.sh status
ZooKeeper JMX enabled by default
Using config: /data/bigdata/src/zookeeper-3.4.12/bin/../conf/zoo.cfg
Mode: follower
node1 zookeeper status done
ZooKeeper JMX enabled by default
Using config: /data/bigdata/src/zookeeper-3.4.12/bin/../conf/zoo.cfg
Mode: leader
node2 zookeeper status done
ZooKeeper JMX enabled by default
Using config: /data/bigdata/src/zookeeper-3.4.12/bin/../conf/zoo.cfg
Mode: follower
node3 zookeeper status done

可以看到一個leader，其它為follower就可以*

2、hadoop:

[root@node1 bigdata]# cd /data/bigdata/src/hadoop-2.7.6/sbin/
[root@node1 sbin]# ./start-all.sh 
This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
Starting namenodes on [node1 node2]
node1: starting namenode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-namenode-node1.out
node2: starting namenode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-namenode-node2.out
node1: starting datanode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-datanode-node1.out
node2: starting datanode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-datanode-node2.out
node5: starting datanode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-datanode-node5.out
node3: starting datanode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-datanode-node3.out
node4: starting datanode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-datanode-node4.out
Starting journal nodes [node1 node2 node3]
node1: starting journalnode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-journalnode-node1.out
node2: starting journalnode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-journalnode-node2.out
node3: starting journalnode, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-journalnode-node3.out
Starting ZK Failover Controllers on NN hosts [node1 node2]
node2: starting zkfc, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-zkfc-node2.out
node1: starting zkfc, logging to /data/bigdata/src/hadoop-2.7.6/logs/hadoop-root-zkfc-node1.out
starting yarn daemons
starting resourcemanager, logging to /data/bigdata/src/hadoop-2.7.6/logs/yarn-root-resourcemanager-node1.out
node1: starting nodemanager, logging to /data/bigdata/src/hadoop-2.7.6/logs/yarn-root-nodemanager-node1.out
node3: starting nodemanager, logging to /data/bigdata/src/hadoop-2.7.6/logs/yarn-root-nodemanager-node3.out
node5: starting nodemanager, logging to /data/bigdata/src/hadoop-2.7.6/logs/yarn-root-nodemanager-node5.out
node2: starting nodemanager, logging to /data/bigdata/src/hadoop-2.7.6/logs/yarn-root-nodemanager-node2.out
node4: starting nodemanager, logging to /data/bigdata/src/hadoop-2.7.6/logs/yarn-root-nodemanager-node4.out
[root@node1 sbin]# jps
16418 QuorumPeerMain
18196 Jps
17047 NameNode
17194 DataNode
17709 DFSZKFailoverController
17469 JournalNode
17999 NodeManager
[root@node1 sbin]#

沒有啟動的去相應(yīng)服務(wù)器下單獨啟動

resourcemanager:需單獨啟動

[root@node3 sbin]# ./yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /data/bigdata/src/hadoop-2.7.6/logs/yarn-root-resourcemanager-node3.out
[root@node3 sbin]# jps
15968 Jps
14264 QuorumPeerMain
14872 NodeManager
14634 DataNode
15723 ResourceManager
14749 JournalNode
[root@node3 sbin]# 

[root@node4 sbin]# ./yarn-daemon.sh start resourcemanager
starting resourcemanager, logging to /data/bigdata/src/hadoop-2.7.6/logs/yarn-root-resourcemanager-node4.out
[root@node4 sbin]# jps
2995 NodeManager
4004 ResourceManager
4091 Jps
2813 DataNode
[root@node4 sbin]#

3、spark

[root@node1 sbin]# cd /data/bigdata/src/spark-2.1.2-bin-hadoop2.7/sbin/
[root@node1 sbin]# ./start-all.sh 
starting org.apache.spark.deploy.master.Master, logging to /data/bigdata/spark/logs/spark-root-org.apache.spark.deploy.master.Master-1-node1.out
node5: starting org.apache.spark.deploy.worker.Worker, logging to /data/bigdata/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-node5.out
node1: starting org.apache.spark.deploy.worker.Worker, logging to /data/bigdata/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-node1.out
node4: starting org.apache.spark.deploy.worker.Worker, logging to /data/bigdata/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-node4.out
node2: starting org.apache.spark.deploy.worker.Worker, logging to /data/bigdata/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-node2.out
node3: starting org.apache.spark.deploy.worker.Worker, logging to /data/bigdata/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-node3.out
[root@node1 sbin]# ./start-history-server.sh 
starting org.apache.spark.deploy.history.HistoryServer, logging to /data/bigdata/spark/logs/spark-root-org.apache.spark.deploy.history.HistoryServer-1-node1.out
[root@node1 sbin]#

4、hbase

[root@node1 ~]# cd /data/bigdata/src/hbase-1.2.6/bin/
[root@node1 bin]# ./start-hbase.sh 
starting master, logging to /data/bigdata/hbase/logs/hbase-root-master-node1.out
node3: starting regionserver, logging to /data/bigdata/hbase/logs/hbase-root-regionserver-node3.out
node2: starting regionserver, logging to /data/bigdata/hbase/logs/hbase-root-regionserver-node2.out
node1: starting regionserver, logging to /data/bigdata/hbase/logs/hbase-root-regionserver-node1.out
[root@node1 bin]#

END

向AI問一下細(xì)節(jié)

大數(shù)據(jù)Hadoop集群搭建

一、環(huán)境

服務(wù)器配置：

操作系統(tǒng)

軟件版本及部署分布：

二、常用命令

1、查看系統(tǒng)基本配置：

2、啟動集群

3、關(guān)閉集群

4、監(jiān)控集群

5、單個進(jìn)程啟動/關(guān)閉

三、 環(huán)境準(zhǔn)備（所有服務(wù)器）

1、設(shè)置主機(jī)名（其它類似）

2、關(guān)閉防火墻firewalld并禁止開機(jī)自啟動和SELINUX

3、修改hosts文件

4、設(shè)置ssh免密登陸,可用ansible

5、修改文件句柄數(shù)

6、時間同步

ntp服務(wù)器設(shè)置

其它主機(jī)設(shè)置,以node2為例

7、上傳安裝包到node1服務(wù)器

8、安裝JDK

四、 安裝zookeeper

1、解壓zookeeper

2、配置zoo.cfg文件

3、添加myid，分發(fā)(安裝個數(shù)為奇數(shù))

4、分發(fā)：

五、 安裝Hadoop

1、解壓hadoop

2、配置hadoop-env.sh

3、配置core-site.xml

4、配置yarn-site.xml

5、配置mapred-site.xml

6、配置hdfs-site.xml

7、配置capacity-scheduler.xml

8、配置slaves

9、修改$HADOOP_HOME/sbin/hadoop-daemon.sh

10、修改$HADOOP_HOME/sbin/yarn-daemon.sh

11、分發(fā)

六、啟動過程

1、啟動zookeeper服務(wù):下面兩種方法選一

(1) 同時開啟所有zookeeper節(jié)點

(2) 集群啟動

(3) 啟動客戶端腳本

2、啟動所有journalnode節(jié)點

3、格式化namenode目錄(主節(jié)點node1)

4、啟動當(dāng)前格式化的namenode進(jìn)程(主節(jié)點node1)

5、在沒有格式化的NN上 執(zhí)行同步命令(副節(jié)點node2)

6、格式化ZKFC

7、啟動hdfs（datanode）

8、啟動yarn：

9、兩臺resourcemanager上啟動resourcemanager

（1）單獨啟動

（2）集群啟動

到目前為止所啟動的進(jìn)程：

10、驗證HDFS的HA功能

七、scala

1、配置前準(zhǔn)備

2、解壓

3、查看scala版本

4、運行scala命令

八、 安裝spark

1、解壓spark

2、配置spark-env.sh

3、配置spark-defaults.conf

4、配置slaves

5、分發(fā)

6、啟動spark

web界面：http://192.168.1.101:8080/

訪問WEBUI： http://192.168.1.101:18080/

到目前為止所啟動的進(jìn)程：

九、 安裝hbase

1、解壓hbase

2、修改$HBASE_HOME/conf/hbase-env.sh,添加

3、修改regionservers文件

4、修改hbase-site.xml文件

5、分發(fā)

6、啟動hbase

到目前為止所啟動的進(jìn)程：

十、kafka

一、環(huán)境

二、常用命令

2、啟動集群

4、監(jiān)控集群

三、環(huán)境準(zhǔn)備（所有服務(wù)器）

2、關(guān)閉防火墻firewalld并禁止開機(jī)自啟動和SELINUX

4、設(shè)置ssh免密登陸,可用ansible

6、時間同步

7、上傳安裝包到node1服務(wù)器

8、安裝JDK

四、安裝zookeeper

1、解壓zookeeper

2、配置zoo.cfg文件

3、添加myid，分發(fā)(安裝個數(shù)為奇數(shù))

4、分發(fā)：

五、安裝Hadoop

2、配置hadoop-env.sh

4、配置yarn-site.xml

5、配置mapred-site.xml

6、配置hdfs-site.xml

8、配置slaves

9、修改$HADOOP_HOME/sbin/hadoop-daemon.sh

六、啟動過程

1、啟動zookeeper服務(wù):下面兩種方法選一

3、格式化namenode目錄(主節(jié)點node1)

4、啟動當(dāng)前格式化的namenode進(jìn)程(主節(jié)點node1)

5、在沒有格式化的NN上執(zhí)行同步命令(副節(jié)點node2)

7、啟動hdfs（datanode）

9、兩臺resourcemanager上啟動resourcemanager

10、驗證HDFS的HA功能

七、scala

1、配置前準(zhǔn)備

2、解壓

3、查看scala版本

4、運行scala命令

八、安裝spark

2、配置spark-env.sh

4、配置slaves

6、啟動spark

九、安裝hbase

1、解壓hbase

4、修改hbase-site.xml文件

5、分發(fā)

6、啟動hbase

十、kafka

2、修改server.properties配置文件：

3、分發(fā)

1、安裝mysql數(shù)據(jù)庫

2、解壓添加環(huán)境變量

4、修改server.properties配置文件：

5、初始化并運行

十二、kylin

2、解壓并創(chuàng)建環(huán)境變量

3、復(fù)制hive的相關(guān)jar到kylin

4、配置Kylin使用的Hive數(shù)據(jù)庫：

5、如果沒有啟動https，請關(guān)閉

6、修改$KYLIN_HOME/bin/kylin.sh

7、檢查kylin依賴

8、啟動kylin服務(wù)

9、配置hive數(shù)據(jù)源

十三、記一次日常啟動

3、spark