[HADOOP] 하둡 KNN은 100 % 0 %를 줄일지도에 붙어 알고리즘을 결합

하둡 KNN은 100 % 0 %를 줄일지도에 붙어 알고리즘을 결합

15/06/11 10시 31분 51초 정보 mapreduce.Job :지도 100 %가 0 %로 감소

내 노트북 ​​(OSX)에 설치 의사 분산 운영 - 나는 KNN은 단일 노드 클러스터에 대한 하둡 2.6.0에 맵리 듀스 hbrj 알고리즘을 결합 오픈 소스를 실행하려합니다. (소스는 여기에서 찾을 수 있습니다 : http://www.cs.utah.edu/~lifeifei/knnj/을). 이 알고리즘은 두 번째 단계는 입력으로서 제 위상의 출력 파일을 사용 MapReduce의 두 단계로 이루어진다. 첫 번째 단계지도와 성공적 감소는 - 나는 또한 출력 파일로 볼 수 있으며, 모든 것이 잘 보인다. 두 번째 단계를 실행하는 경우 그러나, 작업이 감소하거나 심지어 내가 믿는 그 단계에 진입 결코에도 불구하고 성공적으로 완료 할 수 있다고한다.

내가 실행으로 여기에 인쇄됩니다 무엇을 2 단계 (나는 그것이 도움이 될 수 있다는 희망에 모든 것을 포함하고있다)

2015-06-11 10:31:47.526 java[3918:305930] Unable to load realm info from SCDynamicStore
15/06/11 10:31:48 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
15/06/11 10:31:49 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
15/06/11 10:31:49 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
15/06/11 10:31:49 INFO jvm.JvmMetrics: Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
15/06/11 10:31:49 INFO mapred.FileInputFormat: Total input paths to process : 64
15/06/11 10:31:49 INFO mapreduce.JobSubmitter: number of splits:64
15/06/11 10:31:50 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1089761712_0001
15/06/11 10:31:50 INFO mapred.LocalJobRunner: OutputCommitter set in config null
15/06/11 10:31:50 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
15/06/11 10:31:50 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapred.FileOutputCommitter
15/06/11 10:31:50 INFO mapreduce.Job: Running job: job_local1089761712_0001
15/06/11 10:31:50 INFO mapred.LocalJobRunner: Waiting for map tasks
15/06/11 10:31:50 INFO mapred.LocalJobRunner: Starting task: attempt_local1089761712_0001_m_000000_0
15/06/11 10:31:50 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/06/11 10:31:50 INFO mapred.Task:  Using ResourceCalculatorProcessTree : null
15/06/11 10:31:50 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/sasha/hbrj/output/part-00042:0+872
15/06/11 10:31:50 INFO mapred.MapTask: numReduceTasks: 0
15/06/11 10:31:50 INFO mapred.LocalJobRunner: 
15/06/11 10:31:50 INFO mapred.Task: Task:attempt_local1089761712_0001_m_000000_0 is done. And is in the process of committing
15/06/11 10:31:50 INFO mapred.LocalJobRunner: 
15/06/11 10:31:50 INFO mapred.Task: Task attempt_local1089761712_0001_m_000000_0 is allowed to commit now
15/06/11 10:31:50 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1089761712_0001_m_000000_0' to hdfs://localhost:9000/user/sasha/hbrj/output2/_temporary/0/task_local1089761712_0001_m_000000
15/06/11 10:31:50 INFO mapred.MapTask: numReduceTasks: 0
15/06/11 10:31:50 INFO mapred.LocalJobRunner: 
15/06/11 10:31:50 INFO mapred.Task: Task:attempt_local1089761712_0001_m_000000_0 is done. And is in the process of committing
15/06/11 10:31:50 INFO mapred.LocalJobRunner: 
15/06/11 10:31:50 INFO mapred.Task: Task attempt_local1089761712_0001_m_000000_0 is allowed to commit now
15/06/11 10:31:50 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1089761712_0001_m_000000_0' to hdfs://localhost:9000/user/sasha/hbrj/output2/_temporary/0/task_local1089761712_0001_m_000000
15/06/11 10:31:50 INFO mapred.LocalJobRunner: hdfs://localhost:9000/user/sasha/hbrj/output/part-00042:0+872
15/06/11 10:31:50 INFO mapred.Task: Task 'attempt_local1089761712_0001_m_000000_0' done.

때까지 이런 식으로 계속 ...

15/06/11 10:31:51 INFO mapred.LocalJobRunner: Finishing task: attempt_local1089761712_0001_m_000012_0
15/06/11 10:31:51 INFO mapred.LocalJobRunner: Starting task: attempt_local1089761712_0001_m_000013_0
15/06/11 10:31:51 INFO util.ProcfsBasedProcessTree: ProcfsBasedProcessTree currently is supported only on Linux.
15/06/11 10:31:51 INFO mapred.Task:  Using ResourceCalculatorProcessTree : null
15/06/11 10:31:51 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/sasha/hbrj/output/part-00015:0+646
15/06/11 10:31:51 INFO mapred.MapTask: numReduceTasks: 0
15/06/11 10:31:51 INFO mapreduce.Job: Job job_local1089761712_0001 running in uber mode : false
15/06/11 10:31:51 INFO mapreduce.Job:  map 100% reduce 0%
15/06/11 10:31:51 INFO mapred.LocalJobRunner: 
15/06/11 10:31:51 INFO mapred.Task: Task:attempt_local1089761712_0001_m_000013_0 is done. And is in the process of committing
15/06/11 10:31:51 INFO mapred.LocalJobRunner: 
15/06/11 10:31:51 INFO mapred.Task: Task attempt_local1089761712_0001_m_000013_0 is allowed to commit now
15/06/11 10:31:51 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1089761712_0001_m_000013_0' to hdfs://localhost:9000/user/sasha/hbrj/output2/_temporary/0/task_local1089761712_0001_m_000013
15/06/11 10:31:51 INFO mapred.LocalJobRunner: hdfs://localhost:9000/user/sasha/hbrj/output/part-00015:0+646
15/06/11 10:31:51 INFO mapred.Task: Task 'attempt_local1089761712_0001_m_000013_0' done.
15/06/11 10:31:51 INFO mapred.LocalJobRunner: Finishing task: attempt_local1089761712_0001_m_000013_0
15/06/11 10:31:51 INFO mapred.LocalJobRunner: Starting task: attempt_local1089761712_0001_m_000014_0

(마지막 작업 인) 아래와 같이 방식으로 반복하고 작업이 성공적으로 완료 말한다 :

15/06/11 10:31:53 INFO mapred.MapTask: numReduceTasks: 0
15/06/11 10:31:53 INFO mapred.LocalJobRunner: 
15/06/11 10:31:53 INFO mapred.Task: Task:attempt_local1089761712_0001_m_000063_0 is done. And is in the process of committing
15/06/11 10:31:53 INFO mapred.LocalJobRunner: 
15/06/11 10:31:53 INFO mapred.Task: Task attempt_local1089761712_0001_m_000063_0 is allowed to commit now
15/06/11 10:31:53 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1089761712_0001_m_000063_0' to hdfs://localhost:9000/user/sasha/hbrj/output2/_temporary/0/task_local1089761712_0001_m_000063
15/06/11 10:31:53 INFO mapred.LocalJobRunner: hdfs://localhost:9000/user/sasha/hbrj/output/part-00004:0+178
15/06/11 10:31:53 INFO mapred.Task: Task 'attempt_local1089761712_0001_m_000063_0' done.
15/06/11 10:31:53 INFO mapred.LocalJobRunner: Finishing task: attempt_local1089761712_0001_m_000063_0
15/06/11 10:31:53 INFO mapred.LocalJobRunner: map task executor complete.
15/06/11 10:31:54 INFO mapreduce.Job: Job job_local1089761712_0001 completed successfully
15/06/11 10:31:54 INFO mapreduce.Job: Counters: 20
    File System Counters
        FILE: Number of bytes read=96487226
        FILE: Number of bytes written=106993472
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
        HDFS: Number of bytes read=1157797
        HDFS: Number of bytes written=884212
        HDFS: Number of read operations=8576
        HDFS: Number of large read operations=0
        HDFS: Number of write operations=4224
    Map-Reduce Framework
        Map input records=793
        Map output records=793
        Input split bytes=6848
        Spilled Records=0
        Failed Shuffles=0
        Merged Map outputs=0
        GC time elapsed (ms)=42
        Total committed heap usage (bytes)=12124160000
    File Input Format Counters 
        Bytes Read=28599
    File Output Format Counters 
        Bytes Written=21848 

내가 지금까지했던 :

여기 비슷한 질문을 발견 : 하둡 단어 수의 예는 100 % 0 %를 줄일지도에 붙어 주어진 조언의 일부를 따랐다. 특히:

추가 참고 : 나는 출판 되었음 사용하고 작업하도록되어 알고리즘 때문에, 나는 문제가 아마도 코드가 3 세이며, 하둡 0.20.2 버전 작성되었습니다하지만 난 안 추측 사실에서 발생할 수있는 생각 이 너무 확인합니다.

나는이 특정 질문을 이해하지만, 나는 누군가가 올바른 방향으로 날 지점 수 있기를 바랍니다. 난 당신이 유용하게 찾을 수있는 다른 작업을 포함하실 수 있습니다. 어떤 도움을 크게 감사합니다!

해결법

from https://stackoverflow.com/questions/30785814/hadoop-knn-join-algorithm-stuck-at-map-100-reduce-0 by cc-by-sa and MIT license