[HADOOP] Hadoop mapReduce HDFS에 값만 저장하는 방법

Hadoop mapReduce HDFS에 값만 저장하는 방법

나는 이것을 사용하여 중복 줄을 지우고있다.

public class DLines
 {
   public static class TokenCounterMapper extends Mapper<Object, Text, Text, IntWritable>
    {
    private final static IntWritable one = new IntWritable(1);
      private Text word = new Text();
      @Override
      public void map(Object key, Text value, Context context) throws IOException, InterruptedException
       {
           String line=value.toString();
           //int hash_code=line.hashCode();
           context.write(value, one);
       }
   }

public static class TokenCounterReducer extends Reducer<Text, IntWritable, Text, IntWritable> 
 {
        @Override
    public void reduce(Text key, Iterable<IntWritable> values, Context context)throws IOException, InterruptedException 
     {
 public void reduce(Text key, Iterable<IntWritable> values, Context context)throws IOException, InterruptedException 
     {
       int sum = 0;
       for (IntWritable value : values) 
       {   
           sum += value.get();        
       }
       if (sum<2)
           {
             context.write(key,new IntWritable(sum));
           }
      }
      }

키만 hdfs에 저장해야합니다.

해결법

1. ==============================

   1.감속기에서 가치를 요구하지 않으면 NullWritable을 사용하십시오.

   감속기에서 가치를 요구하지 않으면 NullWritable을 사용하십시오.

   당신은 단순히 context.write (key, NullWritable.get ());

   운전 기사에게, 당신은 또한

    job.setMapOutputKeyClass(Text.class);
    job.setMapOutputValueClass(IntWritable.class);
   

   &

    job.setOutputKeyClass(Text.class);
    job.setOutputValueClass(NullWritable.class);
   

2. ==============================

   2.NullWritable 클래스를 사용하여이를 수행 할 수 있습니다.

   NullWritable 클래스를 사용하여이를 수행 할 수 있습니다.

   public class DLines
    {
      public static class TokenCounterMapper extends Mapper<Object, Text, Text, IntWritable>
       {
       private final static IntWritable one = new IntWritable(1);
         private Text word = new Text();
         @Override
         public void map(Object key, Text value, Context context) throws IOException, InterruptedException
          {
              String line=value.toString();
              context.write(value, one);
          }
      }
   
   public static class TokenCounterReducer extends Reducer<Text, IntWritable, Text, NullWritable> 
    {
      NullWritable out = NullWritable.get();
           @Override
       public void reduce(Text key, Iterable<IntWritable> values, Context context)throws IOException, InterruptedException 
          {
          int sum = 0;
          for (IntWritable value : values) 
          {   
              sum += value.get();        
          }
          if (sum<2)
              {
                context.write(key,out);
              }
         }
         }
   

   드라이버 코드

   job.setMapOutputKeyClass(Text.class);
   job.setMapOutputValueClass(IntWritable.class);
   job.setOutputKeyClass(Text.class);
   job.setOutputValueClass(NullWritable.class);
   

   희망이 귀하의 질문에 대한 답변.

from https://stackoverflow.com/questions/23601380/hadoop-mapreduce-how-to-store-only-values-in-hdfs by cc-by-sa and MIT license