[HADOOP] JobEnd 리스너 비는 HDFS 경로에서 소스 파일을 이동 촉발하는 파일을 찾을 수 없습니다 예외가 발생합니다

JobEnd 리스너 비는 HDFS 경로에서 소스 파일을 이동 촉발하는 파일을 찾을 수 없습니다 예외가 발생합니다

스파크 버전 : 2.3

예외:

    java.io.FileNotFoundException: File does not exist: <filename>
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:66)
    at org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:56)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsUpdateTimes(FSNamesystem.java:1932)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocationsInt(FSNamesystem.java:1873)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1853)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1825)
    at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:559)
    at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getBlockLocations(AuthorizationProviderProxyClientProtocol.java:87)
    at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java:363)
    at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1060)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2044)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2038)

    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
    at org.apache.hadoop.ipc.RemoteExc

해결법

1. ==============================

   1.스파크에서 그 버그 https://issues.apache.org/jira/browse/SPARK-24364 메일 스레드; http://mail-archives.apache.org/mod_mbox/spark-issues/201805.mbox/%3cJIRA.13161366.1527070555000.13725.1527070560256@Atlassian.JIRA%3e 수정 : https://github.com/apache/spark/pull/21408/commits/c52d972e4ca09e0ede1bb9e60d3c07f80f605f88

   스파크에서 그 버그 https://issues.apache.org/jira/browse/SPARK-24364 메일 스레드; http://mail-archives.apache.org/mod_mbox/spark-issues/201805.mbox/%3cJIRA.13161366.1527070555000.13725.1527070560256@Atlassian.JIRA%3e 수정 : https://github.com/apache/spark/pull/21408/commits/c52d972e4ca09e0ede1bb9e60d3c07f80f605f88

   수정 된 버전 : 2.3.1 / 2.4.0

from https://stackoverflow.com/questions/58075935/spark-jobend-listner-moving-source-file-from-hdfs-path-causes-file-not-found-exc by cc-by-sa and MIT license