您好,登錄后才能下訂單哦!
目標(biāo):
搭建hadoop+hbase+zoopkeer+hive 開(kāi)發(fā)環(huán)境
安裝環(huán)境:
1、centeros 192.168.1.101
2、 centeros 192.168.1.102
開(kāi)發(fā)環(huán)境:
window +eclipse
一、安裝hadoop集群
1、配置hosts
#vi /etc/hosts
192.168.1.101 master
192.168.1.101 slave1
2、關(guān)閉防火墻:
3、配置ssh 無(wú)密碼訪(fǎng)問(wèn)
slave1
#scp ~/.ssh/slave1.id_rsa.pub master:~/.ssh
master:
測(cè)試:ssh master
ssh slave1
4、安裝hadoop
tar -zxvf hadoop-2.8.0.tar.gz
修改配置文件
修改hadoop-env.sh 增加 export JAVA_HOME
修改mapred-env.sh 增加 export JAVA_HOME
修改yarn-env.sh 增加 export JAVA_HOME
修改 core-site.xml :
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
<description>HDFS address </description>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/usr/hadoop/hadoop-2.8.0/tmp</value>
<description>namenode tmp file </description>
</property>
<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
<description>HDFS address </description>
</property>
修改mapred.site.xml
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>http://master:9001</value>
</property>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>/usr/hadoop/hadoop-2.8.0/mapred/system</value>
</property>
<property>
<name>mapred.local.dir</name>
<value>/usr/hadoop/hadoop-2.8.0/mapred/local</value>
<final>true</final>
</property>
<property>
<name>mapreduce.jobhistory.address</name>
<value>master:10020</value>
</property>
<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>master:19888</value>
</property>
</configuration>
修改 yarn-site.xml
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8032</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8031</value>
</property>
<property>
<name>yarn.resourcemanager.admin.address</name>
<value>master:8033</value>
</property>
<property>
<name>yarn.resourcemanager.webapp.address</name>
<value>master:8088</value>
</property>
修改hdfs-site.xml
<configuration>
<property>
<name>dfs.name.dir</name>
<value>/usr/hadoop/hadoop-2.8.0/hdfs/name</value>
<description>namenode </description>
</property>
<property>
<name>dfs.data.dir</name>
<value>/usr/hadoop/hadoop-2.8.0/hdfs/data</value>
<description>datanode脡</description>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
<description>num</description>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
<description>
true or false
</description>
</property>
</configuration>
建立slaves文件并加入slave1
復(fù)制文件到slave1
scp -r ~/usr/hadoop slave1:~/usr
加入hadoop bin 到環(huán)境變量中
格式化namenode
hadoop namenode -format
啟動(dòng)hadoop
./start-all.sh
檢查服務(wù)啟動(dòng)情況 :jps
master :包含ResourceManager、SecondaryNameNode、NameNode
slave1 :包含datanode NodeManager
下次再說(shuō)zoopker +hbase
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀(guān)點(diǎn)不代表本網(wǎng)站立場(chǎng),如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。