복붙노트

[HADOOP] 하둡으로 Giraph 빌드하기

HADOOP

하둡으로 Giraph 빌드하기

Giado와 Hadoop 2.7.1을 설정하려고합니다. 시도해 볼 수도 있지만 작동하지 않는 것 같습니다.

아래의 링크를 따라 해 보았습니다.

mvn을 사용하여 빌드하려고하면 어떻게됩니까? -Phadoop_yarn -Dhadoop.version = 2.7.1 -DskipTests package

hadoopuser:/usr/local/giraph$ mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests package
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] Apache Giraph Parent
[INFO] Apache Giraph Core
[INFO] Apache Giraph Examples
[INFO] Apache Giraph Distribution
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Parent 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-parent ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-only (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-duplicate (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-dep-mgt (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-duplicate-finder-plugin:1.0.4:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ giraph-parent ---
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Core 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-core ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-core ---
[INFO] 
[INFO] --- munge-maven-plugin:1.0:munge (munge) @ giraph-core ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-core ---
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ giraph-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ giraph-core ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 711 source files to /usr/local/giraph/giraph-core/target/munged/classes
[INFO] -------------------------------------------------------------
[WARNING] COMPILATION WARNING : 
[INFO] -------------------------------------------------------------
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[57,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[39,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Some input files use or override a deprecated API.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Recompile with -Xlint:deprecation for details.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Some input files use unchecked or unsafe operations.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Recompile with -Xlint:unchecked for details.
[INFO] 10 warnings 
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[INFO] 2 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Giraph Parent ............................... SUCCESS [  4.374 s]
[INFO] Apache Giraph Core ................................. FAILURE [  8.469 s]
[INFO] Apache Giraph Examples ............................. SKIPPED
[INFO] Apache Giraph Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 13.136 s
[INFO] Finished at: 2015-11-16T10:19:07+05:30
[INFO] Final Memory: 45M/335M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project giraph-core: Compilation failure: Compilation failure:
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :giraph-cor

mvn을 사용하여 빌드하려고하면 어떻게됩니까? -Phadoop_yarn -Dhadoop.version = 2.7.1 -DskipTests install

hadoopuser:/usr/local/giraph$ mvn -Phadoop_yarn -Dhadoop.version=2.7.1 -DskipTests install
[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] Apache Giraph Parent
[INFO] Apache Giraph Core
[INFO] Apache Giraph Examples
[INFO] Apache Giraph Distribution
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Parent 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-parent ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-parent ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-only (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-duplicate (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-dependency-plugin:2.8:analyze-dep-mgt (default) @ giraph-parent ---
[INFO] Skipping plugin execution
[INFO] 
[INFO] --- maven-duplicate-finder-plugin:1.0.4:check (default) @ giraph-parent ---
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ giraph-parent ---
[INFO] 
[INFO] --- maven-install-plugin:2.3.1:install (default-install) @ giraph-parent ---
[INFO] Installing /usr/local/giraph/pom.xml to /home/hadoopuser/.m2/repository/org/apache/giraph/giraph-parent/1.1.0/giraph-parent-1.1.0.pom
[INFO] Installing /usr/local/giraph/target/giraph-parent-1.1.0-site.xml to /home/hadoopuser/.m2/repository/org/apache/giraph/giraph-parent/1.1.0/giraph-parent-1.1.0-site.xml
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Apache Giraph Core 1.1.0
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- mavanagaiata:0.5.0:branch (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- mavanagaiata:0.5.0:commit (git-commit) @ giraph-core ---
[INFO] 
[INFO] --- maven-enforcer-plugin:1.2:enforce (default) @ giraph-core ---
[INFO] Skipping Rule Enforcement.
[INFO] 
[INFO] --- maven-dependency-versions-check-plugin:2.0.2:check (default) @ giraph-core ---
[INFO] 
[INFO] --- munge-maven-plugin:1.0:munge (munge) @ giraph-core ---
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.4:process (default) @ giraph-core ---
[INFO] 
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ giraph-core ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 2 resources
[INFO] Copying 3 resources
[INFO] 
[INFO] --- maven-compiler-plugin:3.0:compile (default-compile) @ giraph-core ---
[INFO] Changes detected - recompiling the module!
[INFO] Compiling 711 source files to /usr/local/giraph/giraph-core/target/munged/classes
[INFO] -------------------------------------------------------------
[WARNING] COMPILATION WARNING : 
[INFO] -------------------------------------------------------------
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java:[57,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[39,32] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[42,29] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/UnsafeArrayReads.java:[44,25] sun.misc.Unsafe is internal proprietary API and may be removed in a future release
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Some input files use or override a deprecated API.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/DistributedCacheUtils.java: Recompile with -Xlint:deprecation for details.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Some input files use unchecked or unsafe operations.
[WARNING] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/utils/TestGraph.java: Recompile with -Xlint:unchecked for details.
[INFO] 10 warnings 
[INFO] -------------------------------------------------------------
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR : 
[INFO] -------------------------------------------------------------
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
  symbol:   variable SASL_PROPS
  location: class org.apache.hadoop.security.SaslRpcServer
[INFO] 2 errors 
[INFO] -------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Giraph Parent ............................... SUCCESS [  4.360 s]
[INFO] Apache Giraph Core ................................. FAILURE [  7.473 s]
[INFO] Apache Giraph Examples ............................. SKIPPED
[INFO] Apache Giraph Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 12.122 s
[INFO] Finished at: 2015-11-16T10:26:04+05:30
[INFO] Final Memory: 46M/374M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.0:compile (default-compile) on project giraph-core: Compilation failure: Compilation failure:
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyClient.java:[84,68] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] /usr/local/giraph/giraph-core/target/munged/main/org/apache/giraph/comm/netty/SaslNettyServer.java:[105,62] cannot find symbol
[ERROR] symbol:   variable SASL_PROPS
[ERROR] location: class org.apache.hadoop.security.SaslRpcServer
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :giraph-core







---
Version information follows,

Hadoop:  

    hadoopuser:~$ hadoop version
    Hadoop 2.7.1
    Subversion https://git-wip-us.apache.org/repos/asf/hadoop.git -r 15ecc87ccf4a0228f35af08fc56de536e6ce657a
    Compiled by jenkins on 2015-06-29T06:04Z
    Compiled with protoc 2.5.0
    From source with checksum fc0a1a23fc1868e4d5ee7fa2b28a58a
    This command was run using /usr/local/hadoop/share/hadoop/common/hadoop-common-2.7.1.jar

Giraph is version 1.1.0 downloaded from here : http://archive.apache.org/dist/giraph/giraph-1.1.0/giraph-dist-1.1.0-src.tar.bz2

Java

    hadoopuser:~$ java -version
    java version "1.8.0_05"
    Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
    Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)

Maven 

    hadoopuser:~$ mvn -version
    Apache Maven 3.3.3 (7994120775791599e205a5524ec3e0dfe41d4a06; 2015-04-22T17:27:37+05:30)
    Maven home: /opt/apache-maven-3.3.3
    Java version: 1.8.0_05, vendor: Oracle Corporation
    Java home: /usr/lib/jvm/jdk1.8.0/jre
    Default locale: en_IN, platform encoding: UTF-8
    OS name: "linux", version: "3.13.0-68-generic", arch: "amd64", family: "unix"

그의 대답에서 Steve Kaeser가 제안한 것처럼, pom.xlm을 변경했고 변경된 부분은 아래와 같습니다.

  <properties>
<hadoop.version>SET_HADOOP_VERSION_USING_MVN_DASH_D_OPTION</hadoop.version>
    <munge.symbols>PURE_YARN</munge.symbols>
    <!-- TODO: add these checks eventually -->
    <project.enforcer.skip>true</project.enforcer.skip>
    <giraph.maven.dependency.plugin.skip>true</giraph.maven.dependency.plugin.skip>
    <giraph.maven.duplicate.finder.skip>true</giraph.maven.duplicate.finder.skip>
  </properties>

이 작업을 마친 후 mvn -Phadoop_yarn -Dhadoop.version = 2.7.1 -DskipTests를 설치하면 많은 출력이 나오고 실패합니다. 출력의 일부는 아래에 나와 있으며,

[INFO] 
[INFO] --- apache-rat-plugin:0.8:check (default) @ giraph-core ---
[INFO] Exclude: CODE_CONVENTIONS
[INFO] Exclude: **/target/**
[INFO] Exclude: /_bsp/**
[INFO] Exclude: .checkstyle
[INFO] Exclude: .reviewboardrc
[INFO] Exclude: .git/**
[INFO] Exclude: .gitignore
[INFO] Exclude: .arcconfig
[INFO] Exclude: *.patch
[INFO] Exclude: .idea/**
[INFO] Exclude: **/*.iml
[INFO] Exclude: **/*.ipr
[INFO] Exclude: **/test/resources/**
[INFO] Exclude: giraph-gora/conf/*
[INFO] Exclude: giraph-rexster/giraph-kibble/src/main/resources/META-INF/services/com.tinkerpop.rexster.extension.RexsterExtension
[INFO] Exclude: src/site/resources/images/*.svg
[INFO] 
[INFO] >>> findbugs-maven-plugin:2.5.1:check (default) > :findbugs @ giraph-core >>>
[INFO] 
[INFO] --- findbugs-maven-plugin:2.5.1:findbugs (findbugs) @ giraph-core ---
[INFO] Fork Value is true
     [java] The following errors occurred during analysis:
     [java] Nov 17, 2015 3:16:36 PM edu.umd.cs.findbugs.TextUIBugReporter reportAnalysisError
     [java] SEVERE: Error scanning java/util/Comparator for referenced classes
     [java] java.lang.ArrayIndexOutOfBoundsException: 51966
     [java]     at org.objectweb.asm.ClassReader.readUTF8(Unknown Source)
     [java]     at org.objectweb.asm.ClassReader.readClass(Unknown Source)

.
.
.
.
.
.
.
.
[INFO] Reactor Summary:
[INFO] 
[INFO] Apache Giraph Parent ............................... SUCCESS [  4.140 s]
[INFO] Apache Giraph Core ................................. FAILURE [ 37.541 s]
[INFO] Apache Giraph Examples ............................. SKIPPED
[INFO] Apache Giraph Distribution ......................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 41.972 s
[INFO] Finished at: 2015-11-17T15:16:37+05:30
[INFO] Final Memory: 68M/537M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.codehaus.mojo:findbugs-maven-plugin:2.5.1:findbugs (findbugs) on project giraph-core: Execution findbugs of goal org.codehaus.mojo:findbugs-maven-plugin:2.5.1:findbugs failed: Java returned: 1 -> [Help 1]
[ERROR] 

해결법

  1. ==============================

    1.나는 오늘이 문제에 부딪쳤다. 마지막으로 그것을 고치기 위해 두 가지 패치가 필요했습니다.

    나는 오늘이 문제에 부딪쳤다. 마지막으로 그것을 고치기 위해 두 가지 패치가 필요했습니다.

  2. ==============================

    2.방금 전에이 문제가 발생하여 메일 링리스트에 hadoop_yarn 프로필의 munge.symbols에서 STATIC_SASL_SYMBOL을 제거하는 답을 발견했습니다.

    방금 전에이 문제가 발생하여 메일 링리스트에 hadoop_yarn 프로필의 munge.symbols에서 STATIC_SASL_SYMBOL을 제거하는 답을 발견했습니다.

    나는 그 변화를 만든 후에 그 프로파일을 만들 수있었습니다.

    다음은 메일 링리스트 아카이브 링크입니다 : http://mail-archives.apache.org/mod_mbox/giraph-user/201501.mbox/%3C54B17196.4040107@hiro-tan.org%3E

  3. from https://stackoverflow.com/questions/33728752/building-giraph-with-hadoop by cc-by-sa and MIT license