您好,登錄后才能下訂單哦!
這篇文章主要為大家展示了“Eclipse下如何搭建Hadoop2.7.0開(kāi)發(fā)環(huán)境”,內(nèi)容簡(jiǎn)而易懂,條理清晰,希望能夠幫助大家解決疑惑,下面讓小編帶領(lǐng)大家一起研究并學(xué)習(xí)一下“Eclipse下如何搭建Hadoop2.7.0開(kāi)發(fā)環(huán)境”這篇文章吧。
1、下載hadoop插件
http://download.csdn.net/detail/tondayong1981/8680589
2、把插件放到eclipse/plugins目錄下
3、重啟eclipse,配置Hadoop installation directory
如果插件安裝成功,打開(kāi)Windows—Preferences后,在窗口左側(cè)會(huì)有Hadoop Map/Reduce選項(xiàng),點(diǎn)擊此選項(xiàng),在窗口右側(cè)設(shè)置Hadoop安裝路徑。
4、配置Map/Reduce Locations
打開(kāi)Windows—Open Perspective—Other
選擇Map/Reduce,點(diǎn)擊OK
在右下方看到如下圖所示
點(diǎn)擊Map/Reduce Location選項(xiàng)卡,點(diǎn)擊右邊小象圖標(biāo),打開(kāi)Hadoop Location配置窗口:
輸入Location Name,任意名稱(chēng)即可.配置Map/Reduce Master和DFS Mastrer,Host和Port配置成與core-site.xml的設(shè)置一致即可。(貌似Map/Reduce Master 的端口設(shè)置任何數(shù)字都可以?)
點(diǎn)擊"Finish"按鈕,關(guān)閉窗口。
點(diǎn)擊左側(cè)的DFSLocations—>myhadoop(上一步配置的location name),如能看到user,表示安裝成功
如果如下圖所示表示安裝失敗,請(qǐng)檢查Hadoop是否啟動(dòng),以及eclipse配置是否正確。
File—>Project,選擇Map/Reduce Project,輸入項(xiàng)目名稱(chēng)WordCount等。
在WordCount項(xiàng)目里新建class,名稱(chēng)為WordCount,代碼如下:
import java.io.IOException; import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.Mapper; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.util.GenericOptionsParser; public class WordCount { public static class TokenizerMapper extends Mapper<Object, Text, Text, IntWritable>{ private final static IntWritable one = new IntWritable(1); private Text word = new Text(); public void map(Object key, Text value, Context context) throws IOException, InterruptedException { StringTokenizer itr = new StringTokenizer(value.toString()); while (itr.hasMoreTokens()) { word.set(itr.nextToken()); context.write(word, one); } } } public static class IntSumReducer extends Reducer<Text,IntWritable,Text,IntWritable> { private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values,Context context) throws IOException, InterruptedException { int sum = 0; for (IntWritable val : values) { sum += val.get(); } result.set(sum); context.write(key, result); } } public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); String[] otherArgs = new GenericOptionsParser(conf, args).getRemainingArgs(); if (otherArgs.length != 2) { System.err.println("Usage: wordcount <in> <out>"); System.exit(2); } Job job = new Job(conf, "word count"); job.setJarByClass(WordCount.class); job.setMapperClass(TokenizerMapper.class); job.setCombinerClass(IntSumReducer.class); job.setReducerClass(IntSumReducer.class); job.setOutputKeyClass(Text.class); job.setOutputValueClass(IntWritable.class); FileInputFormat.addInputPath(job, new Path(otherArgs[0])); FileOutputFormat.setOutputPath(job, new Path(otherArgs[1])); System.exit(job.waitForCompletion(true) ? 0 : 1); } }
1、在HDFS上創(chuàng)建目錄input
hadoop fs -mkdir /user
hadoop fs -mkdir /user/inhput
2、拷貝本地README.txt到HDFS的input里
hadoop fs -copyFromLocal /opt/hadoop/README.txt /user/input
3、點(diǎn)擊WordCount.java,右鍵,點(diǎn)擊Run As—>Run Configurations,配置運(yùn)行參數(shù),即輸入和輸出文件夾
hdfs://localhost:9000/user/input hdfs://localhost:9000/user/output
點(diǎn)擊Run按鈕,運(yùn)行程序。
4、運(yùn)行完成后,查看運(yùn)行結(jié)果
方法1:
hadoop fs -ls output
可以看到有兩個(gè)輸出結(jié)果,_SUCCESS和part-r-00000
執(zhí)行hadoop fs -cat output/*
方法2:
展開(kāi)DFS Locations,如下圖所示,雙擊打開(kāi)part-r00000查看結(jié)果
以上是“Eclipse下如何搭建Hadoop2.7.0開(kāi)發(fā)環(huán)境”這篇文章的所有內(nèi)容,感謝各位的閱讀!相信大家都有了一定的了解,希望分享的內(nèi)容對(duì)大家有所幫助,如果還想學(xué)習(xí)更多知識(shí),歡迎關(guān)注億速云行業(yè)資訊頻道!
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。