您好,登錄后才能下訂單哦!
這篇文章主要介紹“怎么搭建Spark源碼閱讀環(huán)境”,在日常操作中,相信很多人在怎么搭建Spark源碼閱讀環(huán)境問題上存在疑惑,小編查閱了各式資料,整理出簡單好用的操作方法,希望對大家解答”怎么搭建Spark源碼閱讀環(huán)境”的疑惑有所幫助!接下來,請跟著小編一起來學(xué)習(xí)吧!
1.安裝JDK1.7、Maven3.2.5、Scala2.10.4
2.下載解壓Spark1.4.0
下載地址:
http://mirrors.cnnic.cn/apache/spark/spark-1.4.0/spark-1.4.0.tgz
3.下載安裝 IntelliJ IDEA 14.1.4,給IDEA安裝 Scala 插件,然后導(dǎo)入Spark項(xiàng)目
4.在IDEA中編譯Spark
若編譯時(shí)遇到如下圖所示的錯(cuò)誤:
在Maven視窗點(diǎn)擊“Generate Sources and Update Folders For All Projects”,可解決此問題
在“Generate Sources and Update Folders For All Projects”過程中可能會遇到“Unable to import maven project: See logs for details”錯(cuò)誤;可點(diǎn)擊Help-->Show Log in Exploere,查看錯(cuò)誤日志信息如下:
2015-08-29 22:59:08,909 [ 105808] WARN - #org.jetbrains.idea.maven - embedder class org.jetbrains.idea.maven.project.MavenEmbeddersManager.FOR_FOLDERS_RESOLVE is already used 2015-08-29 22:59:08,916 [ 105815] WARN - ution.rmi.RemoteProcessSupport - Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space 2015-08-29 22:59:08,923 [ 105822] WARN - ution.rmi.RemoteProcessSupport - Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: PermGen space 2015-08-29 22:59:08,923 [ 105822] ERROR - #org.jetbrains.idea.maven - Cannot reconnect. java.lang.RuntimeException: Cannot reconnect. at org.jetbrains.idea.maven.server.RemoteObjectWrapper.perform(RemoteObjectWrapper.java:82) at org.jetbrains.idea.maven.server.MavenEmbedderWrapper.customizeForResolve(MavenEmbedderWrapper.java:64) at org.jetbrains.idea.maven.project.MavenProjectsTree.executeWithEmbedder(MavenProjectsTree.java:1366) at org.jetbrains.idea.maven.project.MavenProjectsTree.resolveFolders(MavenProjectsTree.java:1312) at org.jetbrains.idea.maven.project.MavenProjectsProcessorFoldersResolvingTask.perform(MavenProjectsProcessorFoldersResolvingTask.java:39) at org.jetbrains.idea.maven.project.MavenProjectsProcessor.doProcessPendingTasks(MavenProjectsProcessor.java:134) at org.jetbrains.idea.maven.project.MavenProjectsProcessor.access$100(MavenProjectsProcessor.java:30) at org.jetbrains.idea.maven.project.MavenProjectsProcessor$2.run(MavenProjectsProcessor.java:109) at org.jetbrains.idea.maven.utils.MavenUtil$6.run(MavenUtil.java:441) at com.intellij.openapi.application.impl.ApplicationImpl$8.run(ApplicationImpl.java:400) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) at org.jetbrains.ide.PooledThreadExecutor$1$1.run(PooledThreadExecutor.java:56) Caused by: java.rmi.UnmarshalException: Error unmarshaling return; nested exception is: java.io.EOFException at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:246) at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:161) at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:194) at java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:148) at com.sun.proxy.$Proxy91.set(Unknown Source) at sun.reflect.GeneratedMethodAccessor124.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.intellij.execution.rmi.RemoteUtil.invokeRemote(RemoteUtil.java:124) at com.intellij.execution.rmi.RemoteUtil.access$100(RemoteUtil.java:36) at com.intellij.execution.rmi.RemoteUtil$2$1$1.compute(RemoteUtil.java:105) at com.intellij.execution.rmi.RemoteUtil.executeWithClassLoader(RemoteUtil.java:181) at com.intellij.execution.rmi.RemoteUtil$2$1.invoke(RemoteUtil.java:102) at com.sun.proxy.$Proxy91.set(Unknown Source) at org.jetbrains.idea.maven.server.MavenServerManager.create(MavenServerManager.java:175) at org.jetbrains.idea.maven.server.MavenServerManager.create(MavenServerManager.java:71) at org.jetbrains.idea.maven.server.RemoteObjectWrapper.getOrCreateWrappee(RemoteObjectWrapper.java:41) at org.jetbrains.idea.maven.server.MavenServerManager$5.create(MavenServerManager.java:481) at org.jetbrains.idea.maven.server.MavenServerManager$5.create(MavenServerManager.java:469) at org.jetbrains.idea.maven.server.RemoteObjectWrapper.getOrCreateWrappee(RemoteObjectWrapper.java:41) at org.jetbrains.idea.maven.server.MavenEmbedderWrapper.doCustomize(MavenEmbedderWrapper.java:87) at org.jetbrains.idea.maven.server.MavenEmbedderWrapper.access$000(MavenEmbedderWrapper.java:36) at org.jetbrains.idea.maven.server.MavenEmbedderWrapper$2.execute(MavenEmbedderWrapper.java:67) at org.jetbrains.idea.maven.server.RemoteObjectWrapper.perform(RemoteObjectWrapper.java:76) ... 15 more Caused by: java.io.EOFException at java.io.ObjectInputStream$BlockDataInputStream.peekByte(ObjectInputStream.java:2598) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1318) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at sun.rmi.transport.StreamRemoteCall.executeCall(StreamRemoteCall.java:244) ... 38 more 2015-08-29 22:59:08,924 [ 105823] ERROR - #org.jetbrains.idea.maven - IntelliJ IDEA 14.1.4 Build #IU-141.1532.4 2015-08-29 22:59:08,924 [ 105823] ERROR - #org.jetbrains.idea.maven - JDK: 1.7.0_80 2015-08-29 22:59:08,924 [ 105823] ERROR - #org.jetbrains.idea.maven - VM: Java HotSpot(TM) 64-Bit Server VM 2015-08-29 22:59:08,924 [ 105823] ERROR - #org.jetbrains.idea.maven - Vendor: Oracle Corporation 2015-08-29 22:59:08,924 [ 105823] ERROR - #org.jetbrains.idea.maven - OS: Windows 8 2015-08-29 22:59:08,925 [ 105824] ERROR - #org.jetbrains.idea.maven - Last Action: Maven.UpdateFolders
這種情況應(yīng)該是GFW的原因,換一個(gè)時(shí)間段或者使用代理就可以了
5.在IDE中運(yùn)行master:直接運(yùn)行類org.apache.spark.deploy.master.Master,若報(bào)如下圖所示錯(cuò)誤
修改項(xiàng)目根路徑下的pom.xml,將guava的版本為范圍由provided修改為compile
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<version>14.0.1</version>
<scope>compile</scope>
</dependency>
運(yùn)行成功時(shí),控制臺輸出如下:
6.在IDE中運(yùn)行Worker:運(yùn)行類org.apache.spark.deploy.worker.Worker,并指定參數(shù) --webui-port 8081 spark://192.168.63.1:7077
到此,關(guān)于“怎么搭建Spark源碼閱讀環(huán)境”的學(xué)習(xí)就結(jié)束了,希望能夠解決大家的疑惑。理論與實(shí)踐的搭配能更好的幫助大家學(xué)習(xí),快去試試吧!若想繼續(xù)學(xué)習(xí)更多相關(guān)知識,請繼續(xù)關(guān)注億速云網(wǎng)站,小編會繼續(xù)努力為大家?guī)砀鄬?shí)用的文章!
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場,如果涉及侵權(quán)請聯(lián)系站長郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。