[HADOOP] 자바 1.8과 로컬 Mac에서 하둡을 실행

자바 1.8과 로컬 Mac에서 하둡을 실행

맥 시에라에서 단일 노드 하둡을 실행하려고합니다.

여기 내 구성은 다음과 같습니다

JAVA 버전 https://wiki.apache.org/hadoop/HadoopJavaVersions

my-MacBook-Pro:hadoop me$ java -version
java version "1.8.0_25"
Java(TM) SE Runtime Environment (build 1.8.0_25-b17)
Java HotSpot(TM) 64-Bit Server VM (build 25.25-b02, mixed mode)

HDFS-SITE

<configuration>
<property>
        <name>dfs.replication</name>
        <value>1</value>
</property>
</configuration>

CORE-SITE

<configuration>
<property>
        <name>fs.default.name</name>
        <value>hdfs://localhost:9000</value>
</property>
</configuration>

MAPRED-SITE

<configuration>
<property>
        <name>mapred.job.tracker</name>
        <value>localhost:9010</value>
</property>
</configuration>

하둡-ENV

export JAVA_HOME=/Library/Java/JavaVirtualMachines/jdk1.8.0_25.jdk/Contents/Home
$ bin/hadoop  # displays the usage documentation

여기에 몇 가지 추가 설정은 다음과 같습니다

나는 몇 가지 지침이 권장하는 본 ACTIVE을하지 않았다

#export HADOOP_OPTS="-Djava.net.preferIPv4Stack=true -Dsun.security.krb5.debug=true -Dsun.security.spnego.debug"

하둡-ENV 이미 ACTIVE DEFAULT 다른 두 CONFIGS BY

# Some parts of the shell code may do special things dependent upon
# the operating system.  We have to set this here. See the next
# section as to why....
export HADOOP_OS_TYPE=${HADOOP_OS_TYPE:-$(uname -s)}


# Under certain conditions, Java on OS X will throw SCDynamicStore errors
# in the system logs.
# See HADOOP-8719 for more information.  If one needs Kerberos
# support on OS X, one will want to change/remove this extra bit.
case ${HADOOP_OS_TYPE} in
  Darwin*)
    export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.realm= "
    export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.kdc= "
    export HADOOP_OPTS="${HADOOP_OPTS} -Djava.security.krb5.conf= "
  ;;
esac

내가 네임 노드 형식의 경우 여기에 있습니다 :

네임 노드 서식

My-MacBook-Pro:3.0.0 me$ bin/hdfs namenode -format
WARNING: /usr/local/Cellar/hadoop/3.0.0/libexec/logs does not exist. Creating.

네임 노드를 시작 :

2018-04-09 17:45:15,037 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = My-MacBook-Pro.local/10.5.1.16
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 3.0.0
STARTUP_MSG:   classpath = /usr/ …
STARTUP_MSG:   build = https://git-wip-us.apache.org/repos/asf/hadoop.git -r c25427ceca461ee979d30edd7a4b0f50718e6533; compiled by 'andrew' on 2017-12-08T19:16Z
STARTUP_MSG:   java = 1.8.0_25
************************************************************/
2018-04-09 17:45:15,060 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
2018-04-09 17:45:15,076 INFO namenode.NameNode: createNameNode [-format]
2018-04-09 17:45:15,518 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Formatting using clusterid: CID-f925aa84-648a-478b-9134-77d1766a86cb
…

포맷 :

2018-04-09 17:45:17,121 WARN namenode.NameNode: Encountered exception during format: 
java.io.IOException: Cannot create directory /usr/local/Cellar/hadoop/hdfs/tmp/dfs/name/current
    at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:416)
    at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:579)
    at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:601)
    at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:167)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1169)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1610)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1720)
2018-04-09 17:45:17,128 ERROR namenode.NameNode: Failed to start namenode.

예외:

java.io.IOException: Cannot create directory /usr/local/Cellar/hadoop/hdfs/tmp/dfs/name/current
    2018-04-09 17:45:17,140 INFO namenode.NameNode: SHUTDOWN_MSG: 
    /************************************************************
    SHUTDOWN_MSG: Shutting down NameNode at My-MacBook-Pro.local/10.5.1.16
    ************************************************************/

My-MacBook-Pro:3.0.0 me$ sbin/start-dfs.sh
Starting namenodes on [localhost]

구문 오류 :

localhost: /Users/me/.bashrc: line 2: syntax error near unexpected token `('
localhost: /Users/me/.bashrc: line 2: `==> Searching for a previously deleted formula (in the last month)...'
Starting datanodes
localhost: /Users/me/.bashrc: line 2: syntax error near unexpected token `('
localhost: /Users/me/.bashrc: line 2: `==> Searching for a previously deleted formula (in the last month)...'
Starting secondary namenodes [My-MacBook-Pro.local]
My-MacBook-Pro.local: Warning: Permanently added the ECDSA host key for IP address '10.5.13.173' to the list of known hosts.
My-MacBook-Pro.local: /Users/me/.bashrc: line 2: syntax error near unexpected token `('
My-MacBook-Pro.local: /Users/me/.bashrc: line 2: `==> Searching for a previously deleted formula (in the last month)...'

경고:

2018-04-10 08:28:44,359 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

JPS

My-MacBook-Pro:3.0.0 me$ jps
23201 
46872 SecondaryNameNode
46730 DataNode
47039 Jps

주로 어디에서 문제 해결을 시작 궁금해? 마지막 경고가 문제가 보인다. 내가 HTTP에서 아무것도하지 않는 : // localhost를 : 9000 /에 http : // localhost를 : 9010 /에 http : // localhost를 : 9870 /

해결법

from https://stackoverflow.com/questions/49756742/run-hadoop-locally-on-mac-with-java-1-8 by cc-by-sa and MIT license