[HADOOP] Whirr : 시작 클러스터 후 EC2에서 Hadoop 클러스터에 연결할 수 없음

Whirr : 시작 클러스터 후 EC2에서 Hadoop 클러스터에 연결할 수 없음

Whirr을 처음 사용하고 Whirr을 사용하여 EC2에서 Hadoop 클러스터를 설정하려고합니다 .Cloudera에 대한 자습서를 따랐습니다 https://ccp.cloudera.com/display/CDHDOC/Whirr+Installation

Whirr을 설치하기 전에 Hadoop (0.20.2-cdh3u3)을 설치 한 다음 Whirr (0.5.0-cdh3u3)를 설치합니다.

여기 내 클러스터 설정 파일이 있습니다

whirr.cluster-name=large-cluster
whirr.instance-templates=1 hadoop-jobtracker+hadoop-namenode,1 hadoop-datanode+hadoop-tasktracker
whirr.provider=aws-ec2
whirr.identity=XXXXXXXXXXXXXXX
whirr.credential=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub
whirr.hadoop-install-function=install_cdh_hadoop
whirr.hadoop-configure-function=configure_cdh_hadoop
whirr.hardware-id=m1.large
whirr.image-id=us-east-1/ami-da0cf8b3
whirr.location-id=us-east-1

클러스터 라우 칭은 정상적으로 보입니다.

khiem@master ~ $ whirr launch-cluster --config large-hadoop.properties 
Bootstrapping cluster
Configuring template
Starting 1 node(s) with roles [hadoop-datanode, hadoop-tasktracker]
Configuring template
Starting 1 node(s) with roles [hadoop-jobtracker, hadoop-namenode]
Nodes started: [[id=us-east-1/i-9aa01dfd, providerId=i-9aa01dfd, group=large-cluster, name=null, location=[id=us-east-1a, scope=ZONE, description=us-east-1a, parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null, imageId=us-east-1/ami-da0cf8b3, os=[name=null, family=ubuntu, version=10.04, arch=paravirtual, is64Bit=true, description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml], state=RUNNING, loginPort=22, privateAddresses=[10.196.142.64], publicAddresses=[107.20.64.97], hardware=[id=m1.large, providerId=m1.large, name=null, processors=[[cores=2.0, speed=2.0]], ram=7680, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true], [id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null, type=LOCAL, size=420.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsImage=is64Bit()], loginUser=ubuntu, userMetadata={}]]
Nodes started: [[id=us-east-1/i-0aa31e6d, providerId=i-0aa31e6d, group=large-cluster, name=null, location=[id=us-east-1a, scope=ZONE, description=us-east-1a, parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null, imageId=us-east-1/ami-da0cf8b3, os=[name=null, family=ubuntu, version=10.04, arch=paravirtual, is64Bit=true, description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml], state=RUNNING, loginPort=22, privateAddresses=[10.85.130.43], publicAddresses=[50.17.128.123], hardware=[id=m1.large, providerId=m1.large, name=null, processors=[[cores=2.0, speed=2.0]], ram=7680, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true], [id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null, type=LOCAL, size=420.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsImage=is64Bit()], loginUser=ubuntu, userMetadata={}]]
Authorizing firewall ingress to [Instance{roles=[hadoop-jobtracker, hadoop-namenode], publicIp=50.17.128.123, privateIp=10.85.130.43, id=us-east-1/i-0aa31e6d, nodeMetadata=[id=us-east-1/i-0aa31e6d, providerId=i-0aa31e6d, group=large-cluster, name=null, location=[id=us-east-1a, scope=ZONE, description=us-east-1a, parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null, imageId=us-east-1/ami-da0cf8b3, os=[name=null, family=ubuntu, version=10.04, arch=paravirtual, is64Bit=true, description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml], state=RUNNING, loginPort=22, privateAddresses=[10.85.130.43], publicAddresses=[50.17.128.123], hardware=[id=m1.large, providerId=m1.large, name=null, processors=[[cores=2.0, speed=2.0]], ram=7680, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true], [id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null, type=LOCAL, size=420.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsImage=is64Bit()], loginUser=ubuntu, userMetadata={}]}] on ports [50070, 50030] for [116.96.138.41/32]
Authorizing firewall ingress to [Instance{roles=[hadoop-jobtracker, hadoop-namenode], publicIp=50.17.128.123, privateIp=10.85.130.43, id=us-east-1/i-0aa31e6d, nodeMetadata=[id=us-east-1/i-0aa31e6d, providerId=i-0aa31e6d, group=large-cluster, name=null, location=[id=us-east-1a, scope=ZONE, description=us-east-1a, parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null, imageId=us-east-1/ami-da0cf8b3, os=[name=null, family=ubuntu, version=10.04, arch=paravirtual, is64Bit=true, description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml], state=RUNNING, loginPort=22, privateAddresses=[10.85.130.43], publicAddresses=[50.17.128.123], hardware=[id=m1.large, providerId=m1.large, name=null, processors=[[cores=2.0, speed=2.0]], ram=7680, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true], [id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null, type=LOCAL, size=420.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsImage=is64Bit()], loginUser=ubuntu, userMetadata={}]}] on ports [8020, 8021] for [50.17.128.123/32]
Running configuration script
Configuration script run completed
Running configuration script
Configuration script run completed
Completed configuration of large-cluster
Namenode web UI available at http://ec2-50-17-128-123.compute-1.amazonaws.com:50070
Jobtracker web UI available at http://ec2-50-17-128-123.compute-1.amazonaws.com:50030
Wrote Hadoop site file /home/khiem/.whirr/large-cluster/hadoop-site.xml
Wrote Hadoop proxy script /home/khiem/.whirr/large-cluster/hadoop-proxy.sh
Wrote instances file /home/khiem/.whirr/large-cluster/instances
Started cluster of 2 instances
Cluster{instances=[Instance{roles=[hadoop-datanode, hadoop-tasktracker], publicIp=107.20.64.97, privateIp=10.196.142.64, id=us-east-1/i-9aa01dfd, nodeMetadata=[id=us-east-1/i-9aa01dfd, providerId=i-9aa01dfd, group=large-cluster, name=null, location=[id=us-east-1a, scope=ZONE, description=us-east-1a, parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null, imageId=us-east-1/ami-da0cf8b3, os=[name=null, family=ubuntu, version=10.04, arch=paravirtual, is64Bit=true, description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml], state=RUNNING, loginPort=22, privateAddresses=[10.196.142.64], publicAddresses=[107.20.64.97], hardware=[id=m1.large, providerId=m1.large, name=null, processors=[[cores=2.0, speed=2.0]], ram=7680, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true], [id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null, type=LOCAL, size=420.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsImage=is64Bit()], loginUser=ubuntu, userMetadata={}]}, Instance{roles=[hadoop-jobtracker, hadoop-namenode], publicIp=50.17.128.123, privateIp=10.85.130.43, id=us-east-1/i-0aa31e6d, nodeMetadata=[id=us-east-1/i-0aa31e6d, providerId=i-0aa31e6d, group=large-cluster, name=null, location=[id=us-east-1a, scope=ZONE, description=us-east-1a, parent=us-east-1, iso3166Codes=[US-VA], metadata={}], uri=null, imageId=us-east-1/ami-da0cf8b3, os=[name=null, family=ubuntu, version=10.04, arch=paravirtual, is64Bit=true, description=ubuntu-images-us/ubuntu-lucid-10.04-amd64-server-20101020.manifest.xml], state=RUNNING, loginPort=22, privateAddresses=[10.85.130.43], publicAddresses=[50.17.128.123], hardware=[id=m1.large, providerId=m1.large, name=null, processors=[[cores=2.0, speed=2.0]], ram=7680, volumes=[[id=null, type=LOCAL, size=10.0, device=/dev/sda1, durable=false, isBootDevice=true], [id=null, type=LOCAL, size=420.0, device=/dev/sdb, durable=false, isBootDevice=false], [id=null, type=LOCAL, size=420.0, device=/dev/sdc, durable=false, isBootDevice=false]], supportsImage=is64Bit()], loginUser=ubuntu, userMetadata={}]}], configuration={hadoop.job.ugi=root,root, mapred.job.tracker=ec2-50-17-128-123.compute-1.amazonaws.com:8021, hadoop.socks.server=localhost:6666, fs.s3n.awsAccessKeyId=AKIAIGXAURLAB7CQE77A, fs.s3.awsSecretAccessKey=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, fs.s3.awsAccessKeyId=AZIAIGXIOPLAB7CQE77A, hadoop.rpc.socket.factory.class.default=org.apache.hadoop.net.SocksSocketFactory, fs.default.name=hdfs://ec2-50-17-128-123.compute-1.amazonaws.com:8020/, fs.s3n.awsSecretAccessKey=dWDRq2z0EQegdPrbbL8Dab3eCu98O32r3gOrIbOK}}

또한 프록시를 시작하고 Cloudera 튜토리얼에 따라 로컬 Hadoop 구성을 업데이트했지만 hadoop fs -ls /로 HDFS를 테스트하려고 할 때

터미널이 연결 오류를 인쇄합니다.

12/04/12 11:54:43 WARN conf.Configuration: DEPRECATED: hadoop-site.xml found in the classpath. Usage of hadoop-site.xml is deprecated. Instead use core-site.xml, mapred-site.xml and hdfs-site.xml to override properties of core-default.xml, mapred-default.xml and hdfs-default.xml respectively
12/04/12 11:54:43 INFO security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
12/04/12 11:54:45 INFO ipc.Client: Retrying connect to server: ec2-50-17-128-123.compute-1.amazonaws.com/50.17.128.123:8020. Already tried 0 time(s).
12/04/12 11:54:46 INFO ipc.Client: Retrying connect to server: ec2-50-17-128-123.compute-1.amazonaws.com/50.17.128.123:8020. Already tried 1 time(s).
12/04/12 11:54:48 INFO ipc.Client: Retrying connect to server: ec2-50-17-128-123.compute-1.amazonaws.com/50.17.128.123:8020. Already tried 2 time(s).
12/04/12 11:54:49 INFO ipc.Client: Retrying connect to server: ec2-50-17-128-123.compute-1.amazonaws.com/50.17.128.123:8020. Already tried 3 time(s).

프록시 터미널에서

Running proxy to Hadoop cluster at
ec2-50-17-128-123.compute-1.amazonaws.com. Use Ctrl-c to quit.
Warning: Permanently added 'ec2-50-17-128-123.compute-1.amazonaws.com,50.17.128.123' (RSA) to the list of known hosts. 
channel 2: open failed: connect failed: Connection refused channel 2: open failed: connect failed: Connection refused 
channel 2: open failed: connect failed: Connection refused
channel 2: open failed: connect failed: Connection refused 
channel 2: open failed: connect failed: Connection refused

네임 노드 webUI (50070 포트도 사용할 수 없음), 네임 노드로 ssh 할 수 있지만 네임 노드 내부에는 Hadoop 또는 Java 설치가없는 것 같습니다. 이상한 것입니까?

해결법

from https://stackoverflow.com/questions/10117792/whirr-cannot-connect-to-hadoop-cluster-on-ec2-after-lauch-cluster by cc-by-sa and MIT license