读取HDFS上的txt文件并将其内容复制/写入到本地文件系统上新创建的文件时出错

Question

我正在尝试读取HDFS上的一个文件，并使用以下java程序将该文件的内容复制到新创建的本地文件中。仅供参考，我已经在我的机器上安装了hadoop单节点集群。

HdfsCli.java

package com;

import java.io.BufferedOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;

public class HdfsCli {
public void readFile(String file) throws IOException {

    Configuration conf = new Configuration();
    String hadoopConfPath = "\\opt\\hadoop\\etc\\hadoop\\";
    conf.addResource(new Path(hadoopConfPath + "core-site.xml"));
    conf.addResource(new Path(hadoopConfPath + "hdfs-site.xml"));
    conf.addResource(new Path(hadoopConfPath + "mapred-site.xml"));

    FileSystem fileSystem = FileSystem.get(conf);
    // For the join type of queries, output file in the HDFS has 'r' in it.
    // String type="r";

    Path path = new Path(file);
    if (!fileSystem.exists(path)) {
        System.out.println("File " + file + " does not exists");
        return;
    }

    FSDataInputStream in = fileSystem.open(path);

    String filename = file.substring(file.lastIndexOf('/') + 1,
            file.length());

    OutputStream out = new BufferedOutputStream(new FileOutputStream(
            new File("/home/DAS_Pig/" + filename)));

    byte[] b = new byte[1024];
    int numBytes = 0;
    while ((numBytes = in.read(b)) > 0) {
        out.write(b, 0, numBytes);
    }
    conf.clear();
    in.close();
    out.close();
    fileSystem.close();
}

public static void main(String[] args) throws IOException {
    // TODO Auto-generated method stub
    HDFSClient hc = new HDFSClient();
    hc.readFile("hdfs://localhost:9000//DasData//salaries.txt");

    System.out.println("Successfully Done!");
}

}

但是，当我运行这段代码时，出现了以下错误：

Exception in thread "main" org.apache.hadoop.ipc.RemoteException: Server IPC version 9           cannot communicate with client version 4
    at org.apache.hadoop.ipc.Client.call(Client.java:1066)
    at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
    at com.sun.proxy.$Proxy1.getProtocolVersion(Unknown Source)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:396)
    at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:379)
    at org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:119)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:238)
    at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:203)
    at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:89)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1386)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1404)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:254)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:123)
    at com.HDFSClient.readFile(HDFSClient.java:22)
    at com.HdfsCli.main(HdfsCli.java:57)

我是hadoop开发的新手。有人能指导我解决这个问题吗？谢谢!

Answer 1

服务器和客户端版本不同。看起来服务器版本是4.i，而客户端是3.i。您必须将客户端类路径库升级到服务器版本。

Answer 2

删除项目中包含的所有当前hadoop jars依赖项。下载更新版本的Hadoop。使用新jars配置项目的构建路径。