[HADOOP] 돼지와 하둡 연결 오류

돼지와 하둡 연결 오류

mapreduce 모드에서 돼지를 실행할 때 Connection Refused 오류가 발생합니다.

세부: 나는 tarball (pig-0.14)에서 Pig를 설치하고 bashrc에 classpath를 내 보낸다. 나는 모든 Hadoop (hadoop-2.5) 데몬을 가동시키고있다 (JPS에 의해 확인 됨).

[root@localhost sbin]# jps
2272 Jps
2130 DataNode
2022 NameNode
2073 SecondaryNameNode
2238 NodeManager
2190 ResourceManager

나는 mapreduce 모드에서 돼지를 달리고있다.

[root@localhost sbin]# pig
grunt> file = LOAD '/input/pig_input.csv' USING PigStorage(',') AS (col1,col2,col3);
grunt> dump file;  

그런 다음 오류가 발생합니다.

java.io.IOException: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:334)
    at org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:419)
    at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:532)
    at org.apache.hadoop.mapreduce.Cluster.getJob(Cluster.java:183)
    at org.apache.pig.backend.hadoop.executionengine.shims.HadoopShims.getTaskReports(HadoopShims.java:231)
    at org.apache.pig.tools.pigstats.mapreduce.MRJobStats.addMapReduceStatistics(MRJobStats.java:352)
    at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.addSuccessJobStats(MRPigStatsUtil.java:233)
    at org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil.accumulateStats(MRPigStatsUtil.java:165)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:360)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:280)
    at org.apache.pig.PigServer.launchPlan(PigServer.java:1390)
    at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1375)
    at org.apache.pig.PigServer.storeEx(PigServer.java:1034)
    at org.apache.pig.PigServer.store(PigServer.java:997)
    at org.apache.pig.PigServer.openIterator(PigServer.java:910)
    at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:746)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:372)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:230)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
    at org.apache.pig.Main.run(Main.java:558)
    at org.apache.pig.Main.main(Main.java:170)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:730)
    at org.apache.hadoop.ipc.Client.call(Client.java:1415)
    at org.apache.hadoop.ipc.Client.call(Client.java:1364)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
    at com.sun.proxy.$Proxy15.getJobReport(Unknown Source)
    at org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:133)
    at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:483)
    at org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:320)
    ... 26 more
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:712)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:529)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:493)
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:606)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:700)
    at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:367)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1463)
    at org.apache.hadoop.ipc.Client.call(Client.java:1382)
    ... 34 more  

돼지 설정 파일을 설정하지 않았습니다. 돼지 설정에서 hadoop 설치를 확인하기 위해 변경해야합니까? bashrc에서 HADOOP_HOME이 올바르게 설정되었습니다. 도와주세요.

해결법

1. ==============================

   1.시작하려면 hadoop의 sbin 디렉토리로 이동 한 다음 명령을 입력하십시오.

   시작하려면 hadoop의 sbin 디렉토리로 이동 한 다음 명령을 입력하십시오.

   mr-jobhistory-server.sh start historyserver --config $HADOOP_CONF_DIR
   

   $ HADOOP_CONF_DIR은 hadoop의 config 파일들이 hdfs-site.xml 등과 같은 디렉토리이다.

2. ==============================

   2.해결책을 찾았습니다. 라인 :

   해결책을 찾았습니다. 라인 :

   Caused by: java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to 0.0.0.0:10020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
   

   JobHistoryServer가 작동하지 않을 때 발생합니다. JobHistoryServer를 시작하면이 문제가 해결됩니다. 시작하려면 hadoop의 sbin 디렉토리로 이동 한 다음 명령을 실행하십시오.

   mr-jobhistory-server.sh start
   

   jps를 수행하여 JobHistoryServer가 작동하는지 확인한 다음 돼지 명령을 다시 실행하십시오.

3. ==============================

   3.업데이트 된 명령 (2017 년 기준) :

   업데이트 된 명령 (2017 년 기준) :

   ./mr-jobhistory-daemon.sh --config $HADOOP_CONF_DIR start historyserver
   

   검사

   jps
   

from https://stackoverflow.com/questions/28061100/pig-and-hadoop-connection-error by cc-by-sa and MIT license