如何使用Maven創(chuàng)建Hadoop2項目

本篇內(nèi)容介紹了“如何使用Maven創(chuàng)建Hadoop2項目 ”的有關(guān)知識，在實際案例的操作過程中，不少人都會遇到這樣的困境，接下來就讓小編帶領(lǐng)大家學(xué)習(xí)一下如何處理這些情況吧！希望大家仔細(xì)閱讀，能夠?qū)W有所成！

使用Maven創(chuàng)建Hadoop2項目

    <dependencies>
        <dependency>
            <groupId>jdk.tools</groupId>
            <artifactId>jdk.tools</artifactId>
            <version>1.7</version>
            <scope>system</scope>
            <systemPath>Java_Home/lib/tools.jar</systemPath>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>2.5.2</version>
            <exclusions>
                <exclusion>
                    <groupId>jdk.tools</groupId>
                    <artifactId>jdk.tools</artifactId>
                </exclusion>
            </exclusions>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>2.5.2</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>2.5.2</version>
        </dependency>
    </dependencies>

連接到Hadoop2環(huán)境，例如： 

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.conf.Configuration;

public class IPStarter {
    public static void main(String[] args)  throws Exception {
        Configuration conf = new Configuration();
        //連接到hadoop2集群調(diào)試
        conf.set("fs.defaultFS","hdfs://localhost:9000");
        conf.set("mapreduce.framework.name", "yarn");
        conf.set("yarn.resourcemanager.address", "localhost:8032");
        conf.set("yarn.resourcemanager.scheduler.address", "localhost:8030");
        //conf.set("mapred.remote.os", "Linux");
        Job job = new Job(conf, "JobName");
        job.setJar("/home/grid/study/hadoopDev2/target/hadoopDev2-1.0-SNAPSHOT.jar"); //開發(fā)環(huán)境調(diào)試時使用
        job.setMapperClass(IPMap.class);
        job.setReducerClass(IPReduce.class);

        job.setMapOutputKeyClass(Text.class);
        job.setMapOutputValueClass(Text.class);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);

        FileInputFormat.addInputPath(job, new Path("hdfs://localhost:9000/dev2/in/access.20120104.log"));
        FileOutputFormat.setOutputPath(job, new Path("hdfs://localhost:9000/dev2/out/ip"));
        System.exit(job.waitForCompletion(true) ? 0 : 1);
    }
}

PS：復(fù)制 HADOOP_HOME/etc/hadoop/log4j.properties 到程序的 src 目錄下，這樣調(diào)試時才能在控制臺看到 Job 執(zhí)行的詳細(xì)日志 

“如何使用Maven創(chuàng)建Hadoop2項目 ”的內(nèi)容就介紹到這里了，感謝大家的閱讀。如果想了解更多行業(yè)相關(guān)的知識可以關(guān)注億速云網(wǎng)站，小編將為大家輸出更多高質(zhì)量的實用文章！