dse 4.5에서 스파크 마스터를 실행할 수없고 슬레이브 파일이 없습니다.

DSE 4.5에 5 개의 노드 클러스터가 실행 중입니다. 5 개 노드 중 1 개 노드는 hadoop_enabled 및 spark_enabled이지만 스파크 마스터가 실행되고 있지 않습니다.

 ERROR [Thread-709] 2014-07-02 11:35:48,519 ExternalLogger.java (line 73) SparkMaster: Exception in thread "main" org.jboss.netty.channel.ChannelException: Failed to bind to: /54.xxx.xxx.xxx:7077

누구나 이것에 대한 아이디어가 있습니까 ?? SPARK_LOCAL_IP도 내보내려고했지만 작동하지 않습니다.

DSE 설명서에서 spark-env.sh 구성 파일이 resources / spark / conf / spark-env.sh라고 잘못 언급했습니다. 구성 디렉토리의 실제 경로는 / etc / dse / spark입니다.

conf dir에는 슬레이브도없고 bin dir에는 RUN 파일도 없습니다. 오류가 발생합니다 $ DSE 스파크

 Welcome to
   ____              __
  / __/__  ___ _____/ /__
 _\ \/ _ \/ _ `/ __/  '_/
/___/ .__/\_,_/_/ /_/\_\   version 0.9.1
   /_/

 Using Scala version 2.10.3 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_51)
 Type in expressions to have them evaluated.
 Type :help for more information.
 Creating SparkContext...
 14/07/03 11:37:41 ERROR Remoting: Remoting error: [Startup failed] [
 akka.remote.RemoteTransportException: Startup failed
    at akka.remote.Remoting.akka$remote$Remoting$$notifyError(Remoting.scala:129)
    at akka.remote.Remoting.start(Remoting.scala:194)
    at akka.remote.RemoteActorRefProvider.init(RemoteActorRefProvider.scala:184)
    at akka.actor.ActorSystemImpl._start$lzycompute(ActorSystem.scala:579)
    at akka.actor.ActorSystemImpl._start(ActorSystem.scala:577)
    at akka.actor.ActorSystemImpl.start(ActorSystem.scala:588)
    at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)
    at akka.actor.ActorSystem$.apply(ActorSystem.scala:104)
    at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:96)
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:126)
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:139)
    at shark.SharkContext.<init>(SharkContext.scala:42)
    at shark.SharkEnv$.initWithSharkContext(SharkEnv.scala:90)
    at com.datastax.bdp.spark.SparkILoop.createSparkContext(SparkILoop.scala:41)
    at $line3.$read$$iwC$$iwC.<init>(<console>:10)
    at $line3.$read$$iwC.<init>(<console>:32)
    at $line3.$read.<init>(<console>:34)
    at $line3.$read$.<init>(<console>:38)
    at $line3.$read$.<clinit>(<console>)
    at $line3.$eval$.<init>(<console>:7)
    at $line3.$eval$.<clinit>(<console>)
    at $line3.$eval.$print(<console>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at    sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:772)
    at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1040)
    at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:609)
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:640)
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:604)
    at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:793)
    at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:838)
    at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:750)
    at com.datastax.bdp.spark.SparkILoop$$anonfun$initializeSparkContext$1.apply(SparkILoop.scala:66)
    at com.datastax.bdp.spark.SparkILoop$$anonfun$initializeSparkContext$1.apply(SparkILoop.scala:66)
    at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:258)
    at com.datastax.bdp.spark.SparkILoop.initializeSparkContext(SparkILoop.scala:65)
    at com.datastax.bdp.spark.SparkILoop.initializeSpark(SparkILoop.scala:47)
    at org.apache.spark.repl.SparkILoop$$anonfun$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:908)
    at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:140)
    at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:53)
    at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:102)
    at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:53)
    at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply$mcZ$sp(SparkILoop.scala:925)
    at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:881)
    at org.apache.spark.repl.SparkILoop$$anonfun$process$1.apply(SparkILoop.scala:881)
    at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135)
    at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:881)
    at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:973)
    at com.datastax.bdp.spark.SparkReplMain$.main(SparkReplMain.scala:22)
    at com.datastax.bdp.spark.SparkReplMain.main(SparkReplMain.scala)
 Caused by: org.jboss.netty.channel.ChannelException: Failed to bind to: /54.xx.xx.xx:0
    at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
    at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:391)
    at akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:388)
    at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)
    at scala.util.Try$.apply(Try.scala:161)
    at scala.util.Success.map(Try.scala:206)
    at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
    at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
    at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
 Caused by: java.net.BindException: Cannot assign requested address
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
    at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
]
org.jboss.netty.channel.ChannelException: Failed to bind to: /54.xxx.xxx.xxx.xxx:0
    at org.jboss.netty.bootstrap.ServerBootstrap.bind(ServerBootstrap.java:272)
    at    akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:391)
    at   akka.remote.transport.netty.NettyTransport$$anonfun$listen$1.apply(NettyTransport.scala:388)
    at scala.util.Success$$anonfun$map$1.apply(Try.scala:206)
    at scala.util.Try$.apply(Try.scala:161)
    at scala.util.Success.map(Try.scala:206)
    at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
    at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:235)
    at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.processBatch$1(BatchingExecutor.scala:67)
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply$mcV$sp(BatchingExecutor.scala:82)
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
    at akka.dispatch.BatchingExecutor$Batch$$anonfun$run$1.apply(BatchingExecutor.scala:59)
    at scala.concurrent.BlockContext$.withBlockContext(BlockContext.scala:72)
    at akka.dispatch.BatchingExecutor$Batch.run(BatchingExecutor.scala:58)
    at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:42)
    at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
    at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
    at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
    at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
    at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
Caused by: java.net.BindException: Cannot assign requested address
    at sun.nio.ch.Net.bind0(Native Method)
    at sun.nio.ch.Net.bind(Net.java:444)
    at sun.nio.ch.Net.bind(Net.java:436)
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214)
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
    at org.jboss.netty.channel.socket.nio.NioServerBoss$RegisterTask.run(NioServerBoss.java:193)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.processTaskQueue(AbstractNioSelector.java:366)
    at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:290)
    at org.jboss.netty.channel.socket.nio.NioServerBoss.run(NioServerBoss.java:42)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)

Type in expressions to have them evaluated.
Type :help for more information.

scala>

spark-env.sh

   export SPARK_HOME="/usr/share/dse/spark"
   export SPARK_MASTER_IP=54.xx.xx.xx (public IP)
   export SPARK_MASTER_PORT=7077
   export SPARK_MASTER_WEBUI_PORT=7080
   export SPARK_WORKER_WEBUI_PORT=7081
   export SPARK_WORKER_MEMORY="4g"
   export SPARK_MEM="2g"
   export SPARK_REPL_MEM="2g"
   export SPARK_CONF_DIR="/etc/dse/spark"
   export SPARK_TMP_DIR="$SPARK_HOME/tmp"
   export SPARK_LOG_DIR="$SPARK_HOME/logs"
   export SPARK_LOCAL_IP=54.xx.xx.xx (public IP)
   export SPARK_COMMON_OPTS="$SPARK_COMMON_OPTS -Dspark.kryoserializer.buffer.mb=10 "
   export SPARK_MASTER_OPTS=" -Dspark.deploy.defaultCores=1 -    Dspark.local.dir=$SPARK_TMP_DIR/master -Dlog4j.configuration=file://$SPARK_CONF_DIR/log4j- server.properties -Dspark.log.file=$SPARK_LOG_DIR/master.log "
   export SPARK_WORKER_OPTS=" -Dspark.local.dir=$SPARK_TMP_DIR/worker -Dlog4j.configuration=file://$SPARK_CONF_DIR/log4j-server.properties -Dspark.log.file=$SPARK_LOG_DIR/worker.log "
   export SPARK_EXECUTOR_OPTS=" -Djava.io.tmpdir=$SPARK_TMP_DIR/executor -Dlog4j.configuration=file://$SPARK_CONF_DIR/log4j-executor.properties "
   export SPARK_REPL_OPTS=" -Djava.io.tmpdir=$SPARK_TMP_DIR/repl/$USER "
   export SPARK_APP_OPTS=" -Djava.io.tmpdir=$SPARK_TMP_DIR/app/$USER "

   # Directory to run applications in, which will include both logs and scratch space  (default: SPARK_HOME/work).
   export SPARK_WORKER_DIR="$SPARK_HOME/work"

해결법

==============================
1.SparkConf에 대해 구성된 Akka 원격 호스트 : 포트 및 reference.conf 파일에서 관련 akka 구성을 확인하십시오. 이것은 akka remote 및 host : port와 관련된 akka 시작 충돌과 같이 사용되지만 이미 사용 중입니다. Spark의 akka ActorSystem이 시작될 때 이미 54.xx.xx.xx : 0을 사용하고 있습니다.

SparkConf에 대해 구성된 Akka 원격 호스트 : 포트 및 reference.conf 파일에서 관련 akka 구성을 확인하십시오. 이것은 akka remote 및 host : port와 관련된 akka 시작 충돌과 같이 사용되지만 이미 사용 중입니다. Spark의 akka ActorSystem이 시작될 때 이미 54.xx.xx.xx : 0을 사용하고 있습니다.

from https://stackoverflow.com/questions/24530266/unable-to-run-spark-master-in-dse-4-5-and-slaves-file-is-missing by cc-by-sa and MIT license

'HADOOP' 카테고리의 다른 글

[HADOOP] API를 통해 Hbase 리전 크기 가져 오기 (0)	2019.08.16
[HADOOP] 파이썬을 사용하여 JSON을 HDFS에 저장 (0)	2019.08.16
[HADOOP] Amazon EMR 및 Hive : 외부 테이블에 하위 디렉토리를로드 할 때 "java.io.IOException : 파일이 아님"예외가 발생 함 (0)	2019.08.16
[HADOOP] spark Yarn 모드 spark-submit에서 applicationId를 얻는 방법 (0)	2019.08.16
[HADOOP] 단일 JVM에서 hadoop 멀티 스레드 방식을 실행하는 방법은 무엇입니까? (0)	2019.08.16

복붙노트

[HADOOP] dse 4.5에서 스파크 마스터를 실행할 수없고 슬레이브 파일이 없습니다.

dse 4.5에서 스파크 마스터를 실행할 수없고 슬레이브 파일이 없습니다.

해결법

'HADOOP' 카테고리의 다른 글

티스토리툴바