spark yarn 提交任務(wù)異常怎么辦

這篇文章主要講解了“spark yarn 提交任務(wù)異常怎么辦”，文中的講解內(nèi)容簡單清晰，易于學(xué)習(xí)與理解，下面請大家跟著小編的思路慢慢深入，一起來研究和學(xué)習(xí)“spark yarn 提交任務(wù)異常怎么辦”吧！

Exception in thread "main" java.lang.IllegalStateException: Library directory 'C:\eclipse-workspace\SparkDemo\assembly\target\scala-2.11\jars' does not exist; make sure Spark is built.
	at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248)
	at org.apache.spark.launcher.CommandBuilderUtils.findJarsDir(CommandBuilderUtils.java:342)
	at org.apache.spark.launcher.YarnCommandBuilderUtils$.findJarsDir(YarnCommandBuilderUtils.scala:38)
	at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:545)
	at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:865)
	at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:179)
	at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
	at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:188)
	at org.apache.spark.SparkContext.<init>(SparkContext.scala:501)
	at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
	at my.test.WordCountYarn.main(WordCountYarn.java:34)
[Thread-1] [INFO] [2021-04-12 18:12:44] Shutdown hook called 
[Thread-1] [INFO] [2021-04-12 18:12:44] Deleting directory C:\Users\zengming.zhang\AppData\Local\Temp\spark-ed6b5d87-e618-4375-9ee4-264c0d832447

SparkConf conf = new SparkConf().setAppName("WordCountYarn").setMaster("yarn-client")
                .set("spark.yarn.dist.files", "C:\\eclipse-workspace\\SparkDemo\\src\\main\\resources\\yarn-site.xml")

//添加此參數(shù)解決
                .set("spark.yarn.jars", "*.jars")

https://blog.csdn.net/qq_40651753/article/details/86239630

感謝各位的閱讀，以上就是“spark yarn 提交任務(wù)異常怎么辦”的內(nèi)容了，經(jīng)過本文的學(xué)習(xí)后，相信大家對spark yarn 提交任務(wù)異常怎么辦這一問題有了更深刻的體會，具體使用情況還需要大家實踐驗證。這里是億速云，小編將為大家推送更多相關(guān)知識點的文章，歡迎關(guān)注！