org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy :NoSuchMethodError

Question

以前，我在single node cluster上使用java在hdfs中创建目录，运行得很流畅，但是一旦我创建了集群的多节点，我就收到了这个错误我得到的堆栈跟踪如下所示

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.io.retry.RetryUtils.getDefaultRetryPolicy(Lorg/apache/hadoop/conf/Configuration;Ljava/lang/String;ZLjava/lang/String;Ljava/lang/String;Ljava/lang/Class;)Lorg/apache/hadoop/io/retry/RetryPolicy;
    at org.apache.hadoop.hdfs.NameNodeProxies.createNNProxyWithClientProtocol(NameNodeProxies.java:410)
    at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:316)
    at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:178)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:665)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:601)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2811)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2848)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2830)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
    at CreateDirectory.main(CreateDirectory.java:44)

下面是CreateDirectory类

public static void main(String[] args) throws SQLException, ClassNotFoundException {
        String hdfsUri = "hdfs://localhost:9000/";
       //String dirName = args[0];
        String dirName=null;
      // String filename = args[1];
        String filename;

        if(args.length<=0) dirName = "ekbana"; filename = "text.csv";

        URL url = null;
        BufferedReader in = null;
        FileSystem hdfs = null;
        FSDataOutputStream outStream = null;
        HttpURLConnection conn = null;
        List<Map<String, String>> flatJson;
        Configuration con = new Configuration();
        try {
            url = new URL("http://crm.bigmart.com.np:81/export/export-sales-data.php?sdate=2016-12-01&edate=2016-12-02&key=jdhcvuicx8ruqe9djskjf90ueddishr0uy8v9hbjncvuw0er8idsnv");
        } catch (MalformedURLException ex) {
        }

        try {
            con.set("fs.hdfs.impl", org.apache.hadoop.hdfs.DistributedFileSystem.class.getName());
            con.set("fs.file.impl", org.apache.hadoop.fs.LocalFileSystem.class.getName());
            hdfs = FileSystem.get(URI.create(hdfsUri), con); // this is line 44 
        } catch (IOException e) {
            e.printStackTrace();
        }

        try {
            System.out.println(hdfs.mkdirs(new Path(hdfsUri + "/" + dirName)));
        } catch (IOException e) {
            e.printStackTrace();
        }

许多站点上的解决方案都说我需要hadoop-common，但我已经有了它，但仍然收到此错误。我怀疑重试策略是否与我的设置相关，如果不是，为什么会出现此错误？

Answer 1

添加maven依赖项有帮助：

<dependency>

   <groupId>org.apache.hadoop</groupId>

   <artifactId>hadoop-hdfs</artifactId>

   <version>2.8.1</version>

</dependency>

Answer 2

我高度赞同Yulia的回答，它引导我理解并解决了我所遇到的问题。你可以按照她的建议做，或者添加：

/path/to/hadoop-client/client/hadoop-hdfs-client-<version_number>.jar

添加到您的hadoop-hdfs-client类路径jar，以便在运行时包含。