您好,登錄后才能下訂單哦!
這篇文章主要介紹Hadoop MultipleOutputs如何輸出到多個(gè)文件中,文中介紹的非常詳細(xì),具有一定的參考價(jià)值,感興趣的小伙伴們一定要看完!
Hadoop MultipleOutputs輸出到多個(gè)文件中的實(shí)現(xiàn)方法
1.輸出到多個(gè)文件或多個(gè)文件夾:
驅(qū)動(dòng)中不需要額外改變,只需要在MapClass或Reduce類中加入如下代碼
private MultipleOutputs<Text,IntWritable> mos; public void setup(Context context) throws IOException,InterruptedException { mos = new MultipleOutputs(context); } public void cleanup(Context context) throws IOException,InterruptedException { mos.close(); }
然后就可以用mos.write(Key key,Value value,String baseOutputPath)代替context.write(key, value);
在MapClass或Reduce中使用,輸出時(shí)也會(huì)有默認(rèn)的文件part-m-00*或part-r-00*,不過(guò)這些文件是無(wú)內(nèi)容的,大小為0. 而且只有part-m-00*會(huì)傳給Reduce。
注意:multipleOutputs.write(key, value, baseOutputPath)方法的第三個(gè)函數(shù)表明了該輸出所在的目錄(相對(duì)于用戶指定的輸出目錄)。
如果baseOutputPath不包含文件分隔符“/”,那么輸出的文件格式為baseOutputPath-r-nnnnn(name-r-nnnnn);
如果包含文件分隔符“/”,例如baseOutputPath=“029070-99999/1901/part”,那么輸出文件則為029070-99999/1901/part-r-nnnnn
2.案例-需求
需求,下面是有些測(cè)試數(shù)據(jù),要對(duì)這些數(shù)據(jù)按類目輸出到output中:
1512,iphone5s,4英寸,指紋識(shí)別,A7處理器,64位,M7協(xié)處理器,低功耗 1512,iphone5,4英寸,A6處理器,IOS7 1512,iphone4s,3.5英寸,A5處理器,雙核,經(jīng)典 50019780,ipad,9.7英寸,retina屏幕,豐富的應(yīng)用 50019780,yoga,聯(lián)想,待機(jī)18小時(shí),外形獨(dú)特 50019780,nexus 7,華碩&google,7英寸 50019780,ipad mini 2,retina顯示屏,蘋果,7.9英寸 1101,macbook air,蘋果超薄,OS X mavericks 1101,macbook pro,蘋果,OS X lion 1101,thinkpad yoga,聯(lián)想,windows 8,超級(jí)本
3.Mapper程序:
package cn.edu.bjut.multioutput; import java.io.IOException; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.LongWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Mapper; public class MultiOutPutMapper extends Mapper<LongWritable, Text, IntWritable, Text> { @Override protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { String line = value.toString().trim(); if(null != line && 0 != line.length()) { String[] arr = line.split(","); context.write(new IntWritable(Integer.parseInt(arr[0])), value); } } }
4.Reducer程序:
package cn.edu.bjut.multioutput; import java.io.IOException; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.NullWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Reducer; import org.apache.hadoop.mapreduce.lib.output.MultipleOutputs; public class MultiOutPutReducer extends Reducer<IntWritable, Text, NullWritable, Text> { private MultipleOutputs<NullWritable, Text> multipleOutputs = null; @Override protected void reduce(IntWritable key, Iterable<Text> values, Context context) throws IOException, InterruptedException { for(Text text : values) { multipleOutputs.write("KeySpilt", NullWritable.get(), text, key.toString()+"/"); multipleOutputs.write("AllPart", NullWritable.get(), text); } } @Override protected void setup(Context context) throws IOException, InterruptedException { multipleOutputs = new MultipleOutputs<NullWritable, Text>(context); } @Override protected void cleanup(Context context) throws IOException, InterruptedException { if(null != multipleOutputs) { multipleOutputs.close(); multipleOutputs = null; } } }
5.主程序:
package cn.edu.bjut.multioutput; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.fs.FileSystem; import org.apache.hadoop.fs.Path; import org.apache.hadoop.io.IntWritable; import org.apache.hadoop.io.NullWritable; import org.apache.hadoop.io.Text; import org.apache.hadoop.mapreduce.Job; import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; import org.apache.hadoop.mapreduce.lib.output.MultipleOutputs; import org.apache.hadoop.mapreduce.lib.output.TextOutputFormat; public class MainJob { public static void main(String[] args) throws Exception { Configuration conf = new Configuration(); Job job = new Job(conf, "aaa"); job.setJarByClass(MainJob.class); job.setMapperClass(MultiOutPutMapper.class); job.setMapOutputKeyClass(IntWritable.class); job.setMapOutputValueClass(Text.class); job.setReducerClass(MultiOutPutReducer.class); job.setOutputKeyClass(NullWritable.class); job.setOutputValueClass(Text.class); FileInputFormat.addInputPath(job, new Path(args[0])); MultipleOutputs.addNamedOutput(job, "KeySpilt", TextOutputFormat.class, NullWritable.class, Text.class); MultipleOutputs.addNamedOutput(job, "AllPart", TextOutputFormat.class, NullWritable.class, Text.class); Path outPath = new Path(args[1]); FileSystem fs = FileSystem.get(conf); if(fs.exists(outPath)) { fs.delete(outPath, true); } FileOutputFormat.setOutputPath(job, outPath); job.waitForCompletion(true); } }
以上是“Hadoop MultipleOutputs如何輸出到多個(gè)文件中”這篇文章的所有內(nèi)容,感謝各位的閱讀!希望分享的內(nèi)容對(duì)大家有幫助,更多相關(guān)知識(shí),歡迎關(guān)注億速云行業(yè)資訊頻道!
免責(zé)聲明:本站發(fā)布的內(nèi)容(圖片、視頻和文字)以原創(chuàng)、轉(zhuǎn)載和分享為主,文章觀點(diǎn)不代表本網(wǎng)站立場(chǎng),如果涉及侵權(quán)請(qǐng)聯(lián)系站長(zhǎng)郵箱:is@yisu.com進(jìn)行舉報(bào),并提供相關(guān)證據(jù),一經(jīng)查實(shí),將立刻刪除涉嫌侵權(quán)內(nèi)容。