[HADOOP] 하둡 맵퍼 및 감속기 출력 불일치

하둡 맵퍼 및 감속기 출력 불일치

를 사용하여 다른 매퍼 및 감속기 출력 유형을 구성하려고합니다. setMapOutputKeyClass, setMapOutputValueClass, setMapKeyClass 및 setMapValueClass. 그러나이 함수를 호출 한 후에도 여전히 런타임 중 오류 메시지.

내 코드는 다음과 같습니다.

package org.myorg;

import java.io.IOException;
import java.util.*;

import org.apache.hadoop.fs.Path;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;

public class Sort {

    public static class Map extends MapReduceBase implements Mapper<LongWritable, Text, Text, IntWritable> {
      private final static IntWritable one = new IntWritable(1);
      private Text word = new Text();

      public void map(LongWritable key, Text value, OutputCollector<Text, IntWritable> output, Reporter reporter) throws IOException {
        String line = value.toString();
        StringTokenizer tokenizer = new StringTokenizer(line);
        while (tokenizer.hasMoreTokens()) {
          word.set(tokenizer.nextToken());
          output.collect(word, one);
        }
      }
    }

    public static class Reduce extends MapReduceBase implements Reducer<Text, IntWritable, Text, LongWritable> {
      public void reduce(Text key, Iterator<IntWritable> values, OutputCollector<Text, LongWritable> output, Reporter reporter) throws IOException {
        int sum = 0;
        while (values.hasNext()) {
          sum += values.next().get();
        }
        output.collect(key, new LongWritable(sum));
      }
    }

    public static void main(String[] args) throws Exception {
      JobConf conf = new JobConf(Sort.class);
      conf.setJobName("sort");

      conf.setMapperClass(Map.class);
      conf.setCombinerClass(Reduce.class);
      conf.setReducerClass(Reduce.class);

      conf.setInputFormat(TextInputFormat.class);
      conf.setOutputFormat(TextOutputFormat.class);

      FileInputFormat.setInputPaths(conf, new Path(args[0]));
      FileOutputFormat.setOutputPath(conf, new Path(args[1]));

      conf.setMapOutputKeyClass(Text.class);
      conf.setMapOutputValueClass(IntWritable.class);
      conf.setOutputKeyClass(Text.class);
      conf.setOutputValueClass(LongWritable.class);

      JobClient.runJob(conf);
    }
}

내가 얻은 오류 메시지 :

java.lang.Exception: java.io.IOException: wrong value class: class org.apache.hadoop.io.LongWritable is not class org.apache.hadoop.io.IntWritable
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:354)
Caused by: java.io.IOException: wrong value class: class org.apache.hadoop.io.LongWritable is not class org.apache.hadoop.io.IntWritable
at org.apache.hadoop.mapred.IFile$Writer.append(IFile.java:168)
at org.apache.hadoop.mapred.Task$CombineOutputCollector.collect(Task.java:1160)
at org.myorg.Sort$Reduce.reduce(Sort.java:34)
at org.myorg.Sort$Reduce.reduce(Sort.java:28)
at org.apache.hadoop.mapred.Task$OldCombinerRunner.combine(Task.java:1436)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1441)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1303)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:431)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
13/10/12 14:08:11 INFO mapred.JobClient:  map 0% reduce 0%
13/10/12 14:08:11 INFO mapred.JobClient: Job complete: job_local599611407_0001
13/10/12 14:08:11 INFO mapred.JobClient: Counters: 0
13/10/12 14:08:11 INFO mapred.JobClient: Job Failed: NA
Exception in thread "main" java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
at org.myorg.Sort.main(Sort.java:57)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

내가 잘못 했어? 당신의 도움을 주셔서 감사합니다!

해결법

1. ==============================

   1.아래 줄을 주석으로 처리하면 프로그램이 작동합니다. 다음은 문제가 무엇인지에 대한 설명입니다.

   아래 줄을 주석으로 처리하면 프로그램이 작동합니다. 다음은 문제가 무엇인지에 대한 설명입니다.

   conf.setCombinerClass(Reduce.class);
   

   입력 및 출력 유형이 동일한 감속기를 작성하는 또 다른 솔루션입니다. 이 경우 감속기 클래스를 결합기 클래스로 사용할 수도 있습니다.

from https://stackoverflow.com/questions/19331283/hadoop-mapper-and-reducer-output-mismatch by cc-by-sa and MIT license