[HADOOP] 하둡과 CompositeInputFormat를 사용할 수 없습니다, 던지는 예외 표현은 null입니다

하둡과 CompositeInputFormat를 사용할 수 없습니다, 던지는 예외 표현은 null입니다

나는 CDH4 (4.5)에서 MRv1를 사용 CompositeInputFormat 문제에 직면하고있다. 내가 가입하려고 얼마나 많은 입력 중요하지 않습니다. 간단하게하기 위해, 여기에 하나 개의 입력이 예는 다음과 같습니다

    Configuration conf = new Configuration();

    Job job = new Job(conf, "Blah");
    job.setJarByClass(Blah.class);

    job.setMapperClass(Blah.BlahMapper.class);
    job.setReducerClass(Blah.BlahReducer.class);

    job.setMapOutputKeyClass(LongWritable.class);
    job.setMapOutputValueClass(BlahElement.class);
    job.setOutputKeyClass(LongWritable.class);
    job.setOutputValueClass(BlahElement.class);

    job.setInputFormatClass(CompositeInputFormat.class);
    String joinStatement = CompositeInputFormat.compose("inner", SequenceFileInputFormat.class, "/someinput");
    System.out.println(joinStatement);
    conf.set("mapred.join.expr", joinStatement);
    job.setOutputFormatClass(SequenceFileOutputFormat.class);

    FileOutputFormat.setOutputPath(job, new Path(newoutput));

    return job.waitForCompletion(true) ? 0 : 1;

여기에 출력 +의 스택 트레이스는 다음과 같습니다

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/hadoop2/share/hadoop/mapreduce1/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/hadoop2/share/hadoop/common/lib/slf4j-log4j12-1.6.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
14/01/31 03:27:47 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
inner(tbl(org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat,"/someinput"))
14/01/31 03:27:48 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/01/31 03:27:51 INFO mapred.JobClient: Cleaning up the staging area hdfs://archangel-desktop:54310/tmp/hadoop/mapred/staging/hadoop/.staging/job_201401302213_0013
14/01/31 03:27:51 ERROR security.UserGroupInformation: PriviledgedActionException as:hadoop (auth:SIMPLE) cause:java.io.IOException: Expression is null
Exception in thread "main" java.io.IOException: Expression is null
    at org.apache.hadoop.mapreduce.lib.join.Parser.parse(Parser.java:542)
    at org.apache.hadoop.mapreduce.lib.join.CompositeInputFormat.setFormat(CompositeInputFormat.java:85)
    at org.apache.hadoop.mapreduce.lib.join.CompositeInputFormat.getSplits(CompositeInputFormat.java:127)
    at org.apache.hadoop.mapred.JobClient.writeNewSplits(JobClient.java:1079)
    at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1096)
    at org.apache.hadoop.mapred.JobClient.access$600(JobClient.java:177)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:995)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:948)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:948)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:566)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:596)
    at com.nileshc.graphfu.pagerank.BlockMatVec.run(BlockMatVec.java:79)
    at com.nileshc.graphfu.Main.main(Main.java:21)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

사람은 이전이 직면? 그것을 해결하는 방법에 어떤 아이디어?

해결법

1. ==============================

   1.내 잘못이야.

   내 잘못이야.

   conf.set("mapred.join.expr", joinStatement);
   

   위는해야한다 :

   job.getConfiguration().set("mapreduce.join.expr", joinStatement);
   

   과:

   String joinStatement = CompositeInputFormat.compose("inner", SequenceFileInputFormat.class, "/someinput");
   

   ^^ 그건해야한다 :

   String joinStatement = CompositeInputFormat.compose("inner", SequenceFileInputFormat.class, new Path("/someinput"));
   

   첫 번째 변화는 모든 차이를 만들 것입니다.

2. ==============================

   2.위의 코드에서,

   위의 코드에서,

   conf.set("mapred.join.expr", joinStatement);
   

   작업 개체를 만든 후, 상기 선 부호화된다. 그래서 작업 객체가이 구성을 알고하지 않습니다 분명 !!!!!!

   아래의 수정 된 코드를 참조하십시오 -

   Configuration conf = new Configuration();
   conf.set("mapred.join.expr", joinStatement);
   Job job = new Job(conf, "Blah");
   job.setJarByClass(Blah.class);
   .
   .
   .
   .
   .
   

   다음은 주변의 다른 방법 : -

   job.getConfiguration().set("mapreduce.join.expr", joinStatement);
   

   대신 위의 코드를 사용하여

   conf.set("mapred.join.expr", joinStatement);
   

from https://stackoverflow.com/questions/21468477/cant-use-compositeinputformat-with-hadoop-throwing-exception-expression-is-nul by cc-by-sa and MIT license