[HADOOP] Spark 애플리케이션을 실행하는 동안 java.lang.ClassNotFoundException 가져 오기

Spark 애플리케이션을 실행하는 동안 java.lang.ClassNotFoundException 가져 오기

Spark (Scala)를 처음 사용하고 spark submit을 통해 하나의 spark 응용 프로그램을 실행하려고합니다. 불행히도 java.lang.ClassNotFoundException 예외가 발생합니다.

spark 제출 명령은 다음과 같습니다.

./spark-submit --class "spark.phoenix.a" --master local --deploy-mode client /home/ec2-user/phoenix-0.0.1-SNAPSHOT.jar

내 예외는 다음과 같습니다.

java.lang.ClassNotFoundException: spark.phoenix.a
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:348)
    at org.apache.spark.util.Utils$.classForName(Utils.scala:175)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:689)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

내 샘플 코드는 다음과 같습니다.

package spark.phoenix
import org.apache.spark.SparkConf
import org.apache.spark.sql.SQLContext

object a {

  def main(args: Array[String]) {
    import org.apache.phoenix.spark._
    import org.apache.spark.SparkContext

    val zkQuorum = "localhost:2181"
    val master = "local[*]"
    val sparkConf = new SparkConf()
    sparkConf.setAppName("phoenix-spark-save")
        .setMaster(s"${master}")
    val sc = new SparkContext(sparkConf)
    val sqlContext = new SQLContext(sc)

    // read from orders phoenix table
    val df = sqlContext.phoenixTableAsDataFrame("TABLE1",
        Seq.apply("ID", "COL1"),
        zkUrl = Some.apply(s"${zkQuorum}") )


  }
}

해결법

1. ==============================

   1.문제는 pom.xml 파일에 문제가 있다고 생각합니다 (maven을 사용하는 경우). maven-shade-plugin을 추가하지 않았습니다.

   문제는 pom.xml 파일에 문제가 있다고 생각합니다 (maven을 사용하는 경우). maven-shade-plugin을 추가하지 않았습니다.

   아래 속성을 pom.xml에 추가하고 한 번 시도하십시오.

            <build>
           <resources>
               <resource>
                   <directory>src/main/resources</directory>
               </resource>
           </resources>
           <plugins>
               <plugin>
                   <groupId>net.alchim31.maven</groupId>
                   <artifactId>scala-maven-plugin</artifactId>
                   <version>3.2.2</version>
                   <executions>
                       <execution>
                           <id>eclipse-add-source</id>
                           <goals>
                               <goal>add-source</goal>
                           </goals>
                       </execution>
                       <execution>
                           <id>scala-compile-first</id>
                           <phase>process-resources</phase>
                           <goals>
                               <goal>compile</goal>
                           </goals>
                       </execution>
                       <execution>
                           <id>scala-test-compile-first</id>
                           <phase>process-test-resources</phase>
                           <goals>
                               <goal>testCompile</goal>
                           </goals>
                       </execution>
                       <execution>
                           <id>attach-scaladocs</id>
                           <phase>verify</phase>
                           <goals>
                               <goal>doc-jar</goal>
                           </goals>
                       </execution>
                   </executions>
                   <configuration>
                       <scalaVersion>${scala.version}</scalaVersion>
                       <recompileMode>incremental</recompileMode>
                       <useZincServer>true</useZincServer>
                       <args>
                           <arg>-unchecked</arg>
                           <arg>-deprecation</arg>
                           <arg>-feature</arg>
                       </args>
                       <jvmArgs>
                           <jvmArg>-Xms1024m</jvmArg>
                           <jvmArg>-Xmx1024m</jvmArg>
                           <jvmArg>-XX:ReservedCodeCacheSize=${CodeCacheSize}</jvmArg>
                       </jvmArgs>
                       <javacArgs>
                           <javacArg>-source</javacArg>
                           <javacArg>${java.version}</javacArg>
                           <javacArg>-target</javacArg>
                           <javacArg>${java.version}</javacArg>
                           <javacArg>-Xlint:all,-serial,-path</javacArg>
                       </javacArgs>
                   </configuration>
               </plugin>
               <plugin>
                   <groupId>org.apache.maven.plugins</groupId>
                   <artifactId>maven-shade-plugin</artifactId>
                   <executions>
                       <execution>
                           <phase>package</phase>
                           <goals>
                               <goal>shade</goal>
                           </goals>
                           <configuration>
                               <artifactSet>
                                   <excludes>
                                       <exclude>org.xerial.snappy</exclude>
                                       <exclude>org.scala-lang.modules</exclude>
                                       <exclude>org.scala-lang</exclude>
                                   </excludes>
                               </artifactSet>
                               <filters>
                                   <filter>
                                       <artifact>*:*</artifact>
                                       <excludes>
                                           <exclude>META-INF/*.SF</exclude>
                                           <exclude>META-INF/*.DSA</exclude>
                                           <exclude>META-INF/*.RSA</exclude>
                                       </excludes>
                                   </filter>
                               </filters>
                               <relocations>
                                   <relocation>
                                       <pattern>com.google.common</pattern>
                                       <shadedPattern>shaded.com.google.common</shadedPattern>
                                   </relocation>
                               </relocations>
                           </configuration>
                       </execution>
                   </executions>
               </plugin>
           </plugins>
       </build>
   

2. ==============================

   2.명령에서 ""을 (를) 제거하십시오.

   명령에서 ""을 (를) 제거하십시오.

   이와 같은 것을 시도하십시오.

   ./spark-submit --class spark.phoenix.a --master local --deploy-mode client /home/ec2-user/phoenix-0.0.1-SNAPSHOT.jar
   

from https://stackoverflow.com/questions/42815023/getting-java-lang-classnotfoundexception-while-running-spark-application by cc-by-sa and MIT license