hadoop該怎么部署

這篇文章主要講解了“hadoop該怎么部署”，文中的講解內(nèi)容簡單清晰，易于學習與理解，下面請大家跟著小編的思路慢慢深入，一起來研究和學習“hadoop該怎么部署”吧！

hadoop部署

Hadoop介紹:

廣義: 以apache hadoop軟件為主的生態(tài)圈(hive zookeeper spark hbase)

狹義: 單指apache hadoop軟件

相關(guān)官網(wǎng)：

hadoop.apache.org

hive.apache.org

spark.apache.org

cdh-hadoop：http://archive.cloudera.com/cdh6/cdh/5/hadoop-2.6.0-cdh6.7.0.tar.gz

hadoop軟件及版本:

1.x 企業(yè)不用

2.x 主流

3.x 沒有企業(yè)敢用

a.采坑

b.很多公司都是CDH5.x部署大數(shù)據(jù)環(huán)境 (www.cloudera.com)，即2.6.0-cdh6.7.0 =? apache hadoop2.6.0

很多公司都是CDH5.X部署大數(shù)據(jù)環(huán)境(www.cloudera.com),相當于是把一個生態(tài)圈的組件，集中成為一個系統(tǒng)。

作為基礎(chǔ)環(huán)境，里面裝的2.6.0-cdh6.7.0，注意此版本不等于apache hadoop2.6.0，因為

cdh6.7.0中hadoop做了bug升級。

hadoop軟件:

hdfs：存儲 分布式文件系統(tǒng)

mapreduce:計算。用java計算job1,job2,但企業(yè)不用java(開發(fā)難度大，代碼復(fù)雜)

yarn: 資源和作業(yè)調(diào)度(cpu memory分配)，即：哪個作業(yè)分配到哪個節(jié)點中調(diào)度。

--如果需要按照ssh

Ubuntu Linux:

$ sudo apt-get install ssh

$ sudo apt-get install rsync

----------------------------------------------------------------------------------------------------

安裝部分：

環(huán)境：CentOS 偽分布安裝：即單節(jié)點安裝

HADOOP版本：hadoop-2.6.0-cdh6.7.0.tar.gz

JDK版本：jdk-8u45-linux-x64.gz

安裝原則：不同軟件需要指定對應(yīng)的用戶

linux       root用戶

mysql     mysqladmin用戶

hadoop  hadoop用戶

1.創(chuàng)建hadoop用戶和上傳hadoop軟件

******************************

useradd hadoop

su - hadoop

mkdir app

cd app/

上傳hadoop包

結(jié)果如下：

[hadoop@hadoop app]$ pwd

/home/hadoop/app

[hadoop@hadoop app]$ ls -l

total 304288

drwxr-xr-x 15 hadoop hadoop      4096 Feb 14 23:37 hadoop-2.6.0-cdh6.7.0

-rw-r--r--  1 root   root   311585484 Feb 14 17:32 hadoop-2.6.0-cdh6.7.0.tar.gz

***********************************

2.部署jdk ,要用CDH版本的JDK

***********************************

創(chuàng)建JDK目錄，上傳JDK包要用CDH版本的JDK

su - root

mkdir /usr/java             #上傳JDK包到此目錄

mkdir /usr/share/java   #部署CDH環(huán)境時jdbc jar包需要放到此目錄，否則報錯

cd   /usr/java

tar   -xzvf     jdk-8u45-linux-x64.gz  #解壓JDK

drwxr-xr-x 8 uucp  143      4096 Apr 11  2015 jdk1.8.0_45   #注意解壓后用戶、組是不對的，需要改用戶組為root:root

chown -R root:root jdk1.8.0_45

drwxr-xr-x 8 root root      4096 Apr 11  2015 jdk1.8.0_45

結(jié)果如下：

[root@hadoop java]# pwd

/usr/java

[root@hadoop java]# ll

total 169216

drwxr-xr-x 8 root root      4096 Apr 11  2015 jdk1.8.0_45

-rw-r--r-- 1 root root 173271626 Jan 26 18:35 jdk-8u45-linux-x64.gz

*****************************************

3.設(shè)置java環(huán)境變量

su - root

vi /etc/profile

export JAVA_HOME=/usr/java/jdk1.8.0_45

export JRE_HOME=$JAVA_HOME/jre

export CLASSPATH=.:$JAVA_HOME/lib:$JER_HOME/lib:$CLASSPATH

export PATH=$JAVA_HOME/bin:$JER_HOME/bin:$PATH

source /etc/profile

[root@hadoop java]# which java

/usr/java/jdk1.8.0_45/bin/java

**********************

4.解壓hadoop

su - hadoop

cd  /home/hadoop/app

[hadoop@hadoop002 app]$ tar -xzvf hadoop-2.6.0-cdh6.7.0.tar.gz

[hadoop@hadoop002 app]$ cd hadoop-2.6.0-cdh6.7.0

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$ ll

total 76

drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin  可執(zhí)行腳本

drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 bin-mapreduce1

drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 cloudera

drwxr-xr-x  6 hadoop hadoop  4096 Mar 24  2016 etc  配置目錄(conf)

drwxr-xr-x  5 hadoop hadoop  4096 Mar 24  2016 examples

drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 examples-mapreduce1

drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 include

drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 lib  jar包目錄

drwxr-xr-x  2 hadoop hadoop  4096 Mar 24  2016 libexec

drwxr-xr-x  3 hadoop hadoop  4096 Mar 24  2016 sbin hadoop組件的啟動 停止腳本

drwxr-xr-x  4 hadoop hadoop  4096 Mar 24  2016 share

drwxr-xr-x 17 hadoop hadoop  4096 Mar 24  2016 src

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$

*********************************************************

4.解壓并配置hadoop

su - hadoop

cd app

tar -xzvf hadoop-2.6.0-cdh6.7.0.tar.gz

cd  /home/hadoop/app/hadoop-2.6.0-cdh6.7.0/etc/hadoop

vi  core-site.xml

<configuration>

  <property>

       <name>fs.defaultFS</name>

       <value>hdfs://localhost:9000</value>

   </property>

</configuration>

vi hdfs-site.xml

<configuration>

<property>

       <name>dfs.replication</name>

       <value>1</value>

   </property>

</configuration>

配置hadoop的環(huán)境變量，否則會在啟動時候報錯

vi /home/hadoop/app/hadoop-2.6.0-cdh6.7.0/etc/hadoop/hadoop-env.sh

export HADOOP_CONF_DIR=/home/hadoop/app/hadoop-2.6.0-cdh6.7.0/etc/hadoop

export JAVA_HOME=/usr/java/jdk1.8.0_45

*****************************

*****************************

5.配置ssh localhost無密碼信任關(guān)系

su - hadoop

ssh-keygen  #一直回車

cd .ssh   #可以看到兩個文件

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys  #生成authorized_keys信任文件

ssh localhost date

The authenticity of host 'localhost (127.0.0.1)' can't be established.

RSA key fingerprint is b1:94:33:ec:95:89:bf:06:3b:ef:30:2f:d7:8e:d2:4c.

Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'localhost' (RSA) to the list of known hosts.

Wed Feb 13 22:41:17 CST 2019

chmod 600 authorized_keys   #非常重要，如果不更改權(quán)限，執(zhí)行ssh localhost date時會讓輸入密碼，但hadoop用戶根本無密碼，此時就是權(quán)限搞的貓膩。

**********************************

6.格式化

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$ bin/hdfs namenode -format

***************************************

cd /home/hadoop/app/hadoop-2.6.0-cdh6.7.0

bin/hdfs namenode -format  #為何進入bin 再 hdfs namenode -format說找不到hdfs命令

***************************************

7.啟動hadoop服務(wù)

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$ sbin/start-dfs.sh

19/02/13 22:47:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting namenodes on [localhost]

localhost: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh6.7.0/logs/hadoop-hadoop-namenode-hadoop002.out

localhost: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh6.7.0/logs/hadoop-hadoop-datanode-hadoop002.out

Starting secondary namenodes [0.0.0.0]

The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.

RSA key fingerprint is b1:94:33:ec:95:89:bf:06:3b:ef:30:2f:d7:8e:d2:4c.

Are you sure you want to continue connecting (yes/no)? yes  #輸入yes,,因為ssh 信任關(guān)系 是配置的是localhost，而非0.0.0.0

0.0.0.0: Warning: Permanently added '0.0.0.0' (RSA) to the list of known hosts.

0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh6.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop002.out

19/02/13 22:49:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$ sbin/stop-dfs.sh

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$ sbin/start-dfs.sh

19/02/13 22:57:23 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Starting namenodes on [localhost]

localhost: starting namenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh6.7.0/logs/hadoop-hadoop-namenode-hadoop002.out

localhost: starting datanode, logging to /home/hadoop/app/hadoop-2.6.0-cdh6.7.0/logs/hadoop-hadoop-datanode-hadoop002.out

Starting secondary namenodes [0.0.0.0]

0.0.0.0: starting secondarynamenode, logging to /home/hadoop/app/hadoop-2.6.0-cdh6.7.0/logs/hadoop-hadoop-secondarynamenode-hadoop002.out

19/02/13 22:57:38 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$ jps  #檢驗是否正常啟動，需啟動以下四個服務(wù)

15059 Jps

14948 SecondaryNameNode 第二名稱節(jié)點 老二

14783 DataNode  數(shù)據(jù)節(jié)點  小弟

14655 NameNode  名稱節(jié)點  老大 讀寫

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$

open http://ip:50070  #安裝成功可以打開hadoop的web管理界面:如圖

8.配置hadoop命令環(huán)境變量

***************************************************************

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$ cat ~/.bash_profile

# .bash_profile

# Get the aliases and functions

if [ -f ~/.bashrc ]; then

. ~/.bashrc

fi

# User specific environment and startup programs

export HADOOP_PREFIX=/home/hadoop/app/hadoop-2.6.0-cdh6.7.0

export PATH=$HADOOP_PREFIX/bin:$PATH

source ~/.bash_profile

/home/hadoop/app/hadoop-2.6.0-cdh6.7.0

***************************************************************

9.操作hadoop, hdfs dfs操作命令和Linux命令極其相似

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$ bin/hdfs dfs -ls /

19/02/13 23:08:35 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$ bin/hdfs dfs -ls /

19/02/13 23:11:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$ ls /

bin   dev  home  lib64       media  opt   root  sbin     srv  tmp  var

boot  etc  lib   lost+found  mnt    proc  run   selinux  sys  usr

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$ bin/hdfs dfs -mkdir /ruozedata

19/02/13 23:11:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$ bin/hdfs dfs -ls /

19/02/13 23:11:56 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Found 1 items

drwxr-xr-x   - hadoop supergroup          0 2019-02-13 23:11 /ruozedata

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$ ls /

bin   dev  home  lib64       media  opt   root  sbin     srv  tmp  var

boot  etc  lib   lost+found  mnt    proc  run   selinux  sys  usr

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$

10.查看幫助

[hadoop@hadoop002 hadoop-2.6.0-cdh6.7.0]$  bin/hdfs --help

作業(yè):

1.ssh博客 閱讀 摘抄

http://blog.itpub.net/30089851/viewspace-1992210/

http://blog.itpub.net/30089851/viewspace-2127102/

2.部署hdfs偽分布式

3.博客要寫到hdfs偽分布式

小提示：

如果 su - zookeeper不能切換

解決方法：

更改：/etc/passwd中zookeeper用戶的登錄方式由/sbin/nologin==>/bin/bash即可

感謝各位的閱讀，以上就是“hadoop該怎么部署”的內(nèi)容了，經(jīng)過本文的學習后，相信大家對hadoop該怎么部署這一問題有了更深刻的體會，具體使用情況還需要大家實踐驗證。這里是億速云，小編將為大家推送更多相關(guān)知識點的文章，歡迎關(guān)注！