Spark學(xué)習(xí)之第一個(gè)程序打包、提交任務(wù)到集群

1、免秘鑰登錄配置：

ssh-keygen
cd .ssh
touch authorized_keys
cat id_rsa.pub > authorized_keys
chmod 600 authorized_keys

2、環(huán)境工具

2.1環(huán)境

系統(tǒng) urbuntu jdk 1.7.0_79

scala 2.10.4

hadoop 2.6.0

spark 1.6.2

2.2打包工具

IDEA + sbt1.2打包工具

3.打包

3.1安裝插件

需要預(yù)先安裝scala插件，點(diǎn)擊File ->Setting ->Plugins ->輸入框輸入scala->install
安裝完成需要重啟IDE

3.2創(chuàng)建項(xiàng)目

File -> New Project ->Scala -> SBT 選擇相應(yīng)版本 ->finish

3.3編寫(xiě)代碼

build.sbt 添加spark相關(guān)依賴

name := "demoPro"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "1.6.2"

創(chuàng)建WordCount.scala，編寫(xiě)如下代碼

import org.apache.spark.{SparkContext, SparkConf}

/**
 * Created by Administrator on 2018/2/20.
 */
object WordCount {

  def main(args: Array[String]) {
    val conf = new SparkConf().setAppName("wordcount")
    val sc = new SparkContext(conf)
    val input = sc.textFile("/home/dell/helloSpark.txt")
    val lines = input.flatMap(line => (line.split(" ")))
    val count = lines.map(word => (word, 1)).reduceByKey { case (x, y) => x + y }
    val output=count.saveAsTextFile("/home/dell/helloSparkRes")
  }
}

3.4打包

File -> Project Structure -> Aritifacts -> 點(diǎn)擊+號(hào) ->jar -> 第二個(gè) -> 指定Module和 MainClass -> JAR files from libraries 選擇第二個(gè) ->點(diǎn)擊ok

主題欄點(diǎn)擊Build -> Build Aritifacts - Build

在工程目下out目錄中生成相應(yīng)jar包即打包成功

4.提交任務(wù)

4.1啟動(dòng)hadoop

#進(jìn)入sbin目錄
cd $Hadoop_HOME/sbin
#啟動(dòng)hadoop集群
start-all.sh

4.2上傳測(cè)試文件到hdfs

hadoop fs -put test.txt /test/test.txt

4.3上傳程序jar包

是同filelize 或者sftp 或者 rz -y命令上傳程序jar

4.4 提交任務(wù)

4.4.1啟動(dòng)Master

sudo ./start-master.sh
訪問(wèn)localhost:8080 獲取spark://xxx:7077

4.4.2啟動(dòng)Worker

sudo ./bin/spark-class org.apache.spark.deploy.worker.Worker spark://dell:7077

4.4.3提交作業(yè)

sudo ./bin/spark-submit --master spark://dell:7077 --class WordCount /home/dell/demopro.jar

5、查看測(cè)試程序是否正確

5.1、查看 是否生成文件夾 進(jìn)入文件查看程序是否正確

5.2、進(jìn)入文件查看程序是否正確