HDFS FileSystem关闭异常

Question

我刚刚运行了一个hdfs演示，如下所示：

public final class HDFSRemoveDemo {

    public static void main(String[] args) throws Exception {
        Path root = new Path("hdfs://localhost:49000/");
        FileSystem fs = root.getFileSystem(new Configuration());

        fs.create(new Path("/tmp/test"));
        fs.delete(new Path("/tmp/test"), false);
        fs.close();
    }
}

抛出一个令人费解的异常，如下所示：

org.apache.hadoop.hdfs.DFSClient closeAllFilesBeingWritt
en
SEVERE: Failed to close file /tmp/test
org.apache.hadoop.ipc.RemoteException: org.apache.hadoop.hdfs.server.namenode.Le
aseExpiredException: No lease on /tmp/test File does not exist. Holder DFSClient
_NONMAPREDUCE_-1727094995_1 does not have any open files
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.
java:1999)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.
java:1990)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFileInternal(FSN
amesystem.java:2045)
    at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesyste
m.java:2033)
    at org.apache.hadoop.hdfs.server.namenode.NameNode.complete(NameNode.java:805)
    at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:587)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1432)
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1428)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.ja
va:1190)
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1426)

    at org.apache.hadoop.ipc.Client.call(Client.java:1113)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)
    at com.sun.proxy.$Proxy1.complete(Unknown Source)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57
)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl
.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocati
onHandler.java:85)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHand
ler.java:62)
    at com.sun.proxy.$Proxy1.complete(Unknown Source)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.jav
a:4121)
    at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:4022)
    at org.apache.hadoop.hdfs.DFSClient.closeAllFilesBeingWritten(DFSClient.java:41
7)
    at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:433)
    at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.jav
a:369)

当我移除fs.close();时，它工作得很好。

环境是：

hadoop-core -- 1.2.1

jdk -- 1.6.0_21

当文件系统关闭时发生了什么？有没有人遇到过这个问题？

Answer 1

通常，在执行FileSystem.get(...)时不应调用fs.close()。

FileSystem.get(...)实际上不会打开一个“新的”FileSystem对象。当您在该FileSystem上执行close()时，您也将为任何上游进程关闭它。

例如，如果在映射器过程中关闭FileSystem，则MapReduce驱动程序在清理时再次尝试关闭FileSystem时将失败。