하둡 단어 수를 실행할 수 없음-하둡 오류 메시지를 디코딩하는 데 도움이 필요합니다

직업이 실패한 이유를 알아내는 데 도움이 필요합니다. 나는 하나를 구축 그것을 시도하기 위해 노드 클러스터. 나는 여기 예제를 따랐다.

모든 것이 올바르게 작동하는 것 같습니다. 네임 노드를 포맷하고 웹 인터페이스를 통해 작업 추적기, 데이터 노드 및 네임 노드에 연결할 수 있습니다. 모든 hadoop 서비스를 시작하고 중지 할 수 있습니다.

그러나 wordcount 예제를 실행하려고하면 다음과 같은 결과가 나타납니다.

Error initializing attempt_201105161023_0002_m_000011_0:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161023_0002/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161023_0002/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

로컬 파일 시스템에 디렉토리를 작성했습니다.

$ sudo mkdir /app/hadoop/tmp 
$ sudo chown hadoop:hadoop /app/hadoop/tmp

core-site.xml도 수정했습니다.

<!-- In: conf/core-site.xml --> 
<property> 
  <name>hadoop.tmp.dir</name> 
  <value>/app/hadoop/tmp</value> 
  <description>A base for other temporary directories.</description> 
</property>

네임 노드를 포맷하면 두 로컬 모두에 하위 디렉토리가 생성됩니다. 그리고 HDFS.

실패한 단어 수 출력 결과를 보면 오류가 발생합니다. 메시지가 파일의 I / O 오류에 대해 불평합니다 / app / hadoop / tmp / mapred / local / ttprivate / taskTracker / hadoop / jobcache / job_201105161023_0002 / jobToken

문제를 해결했으며이 jobToken 파일을 찾아 볼 수 있습니다. 로컬 파일 시스템, 문제 없습니다. 내용은 HDTS MapReduce.job 201105161023_0002와 같은 것입니다.

그렇다면 권한 문제입니까? hadoop 프로세스 사용자가 모든 하위 디렉토리에 쓸 수 있도록 만들고 파일을 만들 수있었습니다. 그렇다면 무엇이 더 잘못 될 수 있습니까?

커맨드 라인에서 좀 더 자세한 출력 :

hadoop@jonathan-VirtualBox:/usr/local/hadoop/hadoop-0.20.203.0$ bin/hadoop jar hadoop-examples-0.20.203.0.jar wordcount app/download app/output4
11/05/16 13:38:56 INFO input.FileInputFormat: Total input paths to process : 3
11/05/16 13:39:05 INFO mapred.JobClient: Running job: job_201105161222_0003
11/05/16 13:39:06 INFO mapred.JobClient:  map 0% reduce 0%
11/05/16 13:39:17 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000004_0, Status : FAILED
Error initializing attempt_201105161222_0003_m_000004_0:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:21 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_0&filter=stdout
11/05/16 13:39:21 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_0&filter=stderr
11/05/16 13:39:22 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000004_1, Status : FAILED
Error initializing attempt_201105161222_0003_m_000004_1:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:22 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_1&filter=stdout
11/05/16 13:39:22 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_1&filter=stderr
11/05/16 13:39:23 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000004_2, Status : FAILED
Error initializing attempt_201105161222_0003_m_000004_2:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:23 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_2&filter=stdout
11/05/16 13:39:23 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000004_2&filter=stderr
11/05/16 13:39:30 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_0, Status : FAILED
Error initializing attempt_201105161222_0003_m_000003_0:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:31 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_0&filter=stdout
11/05/16 13:39:31 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_0&filter=stderr
11/05/16 13:39:33 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_1, Status : FAILED
Error initializing attempt_201105161222_0003_m_000003_1:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:33 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_1&filter=stdout
11/05/16 13:39:33 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_1&filter=stderr
11/05/16 13:39:36 INFO mapred.JobClient: Task Id : attempt_201105161222_0003_m_000003_2, Status : FAILED
Error initializing attempt_201105161222_0003_m_000003_2:
java.io.IOException: Exception reading file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:135)
    at org.apache.hadoop.mapreduce.security.TokenCache.loadTokens(TokenCache.java:163)
    at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1064)
    at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1001)
    at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2161)
    at org.apache.hadoop.mapred.TaskTracker$TaskLauncher.run(TaskTracker.java:2125)
Caused by: java.io.FileNotFoundException: File file:/app/hadoop/tmp/mapred/local/ttprivate/taskTracker/hadoop/jobcache/job_201105161222_0003/jobToken does not exist.
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:371)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:245)
    at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
    at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
    at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
    at org.apache.hadoop.security.Credentials.readTokenStorageFile(Credentials.java:129)
    ... 5 more

11/05/16 13:39:36 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_2&filter=stdout
11/05/16 13:39:36 WARN mapred.JobClient: Error reading task outputhttp://jonathan-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201105161222_0003_m_000003_2&filter=stderr
11/05/16 13:39:39 INFO mapred.JobClient: Job complete: job_201105161222_0003
11/05/16 13:39:39 INFO mapred.JobClient: Counters: 4
11/05/16 13:39:39 INFO mapred.JobClient:   Job Counters 
11/05/16 13:39:39 INFO mapred.JobClient:     SLOTS_MILLIS_MAPS=0
11/05/16 13:39:39 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
11/05/16 13:39:39 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
11/05/16 13:39:39 INFO mapred.JobClient:     SLOTS_MILLIS_REDUCES=0

해결법

==============================
1./ etc / hosts 구성을 확인하십시오. 마스터 도메인 이름과 슬레이브에 대해서도 언급해야합니다.

/ etc / hosts 구성을 확인하십시오. 마스터 도메인 이름과 슬레이브에 대해서도 언급해야합니다.

예를 들어 : 127.0.0.1 localhost.localdomain 로컬 호스트 107.108.206.64 마스터 ABC- 우분투 107.108.208.24 슬레이브 XYZ- 우분투 107.108.87.81 슬레이브 1 QWE- 우분투

감소 처리 중에는 슬레이브 대신 XYZ- 우분투 호스트 이름을 검색합니다.

또한 호스트 구성은 모두 동일해야합니다.
==============================
2./ app / hadoop / tmp 폴더를 만들지 마십시오 / tmp / hadoop에서 기본적으로 생성됩니다 ...

/ app / hadoop / tmp 폴더를 만들지 마십시오 / tmp / hadoop에서 기본적으로 생성됩니다 ...

따라서 conf 파일에서 해당 속성을 제거하십시오. 작동합니다

from https://stackoverflow.com/questions/6027325/hadoop-wordcount-unable-to-run-need-help-on-decoding-the-hadoop-error-message by cc-by-sa and MIT license

'HADOOP' 카테고리의 다른 글

[HADOOP] Oozie 작업을 병렬로 실행 (0)	2019.08.15
[HADOOP] 하둡 맵퍼 및 감속기 출력 불일치 (0)	2019.08.15
[HADOOP] HBase : HBase 테이블의 크기를 어떻게 추정 할 수 있습니까? (0)	2019.08.15
[HADOOP] Pig Latin에서 그룹당 하나의 파일 작성 (0)	2019.08.15
[HADOOP] 분산 캐시 hadoop을 통해 jar 추가 (0)	2019.08.15

복붙노트

[HADOOP] 하둡 단어 수를 실행할 수 없음-하둡 오류 메시지를 디코딩하는 데 도움이 필요합니다

하둡 단어 수를 실행할 수 없음-하둡 오류 메시지를 디코딩하는 데 도움이 필요합니다

해결법

1./ etc / hosts 구성을 확인하십시오. 마스터 도메인 이름과 슬레이브에 대해서도 언급해야합니다.

2./ app / hadoop / tmp 폴더를 만들지 마십시오 / tmp / hadoop에서 기본적으로 생성됩니다 ...

'HADOOP' 카테고리의 다른 글

티스토리툴바