集群的Hadoop2客户端

Question

我们有一个linux集群，但是由于各种原因，一些windows客户端将数据连接并将数据推送到linux集群。在hadoop1中，我们能够通过cygwin运行hadoop，但是在hadoop2中，正如网站中所述，并不需要或不支持网站。

问题

1. 究竟发生了什么变化？为什么一个客户(仅)不会在cygwin下运行，或者它可以运行？除了路径之外，还有什么其他考虑因素在起作用？
2. 除了下面用于提交作业的属性之外，对于windows/客户端与linux集群的交互，还有什么需要考虑的吗？ conf.set("mapreduce.app-submission.cross-platform", "true");
3. 提取hadoop-2.6.0-cdh5.5.2并使用$HADOOP_HOME/etc下的正确配置从cygwin运行它会产生一些类路径或类路径形成问题类而不是找到问题？例如，以下运行 hdfs dfs -ls错误:找不到或加载主类org.apache.hadoop.fs.FsShell

然后查看类路径，看起来它们包含了cygwin路径。尝试将它们转换为windows路径，以便查找jar。

in $HADOOP_HOME/etc/hdfs.sh locate the dfs command and change to 
      elif [ "$COMMAND" = "dfs" ] ; then
      if $cygwin; then
         CLASSPATH=`cygpath -p -w "$CLASSPATH"`
      fi
      CLASS=org.apache.hadoop.fs.FsShell

这样做的结果如下：

16/04/07 16:01:05 ERROR util.Shell: Failed to locate the winutils binary in the hadoop binary path
    java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
            at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:378)
            at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:393)
            at org.apache.hadoop.util.Shell.<clinit>(Shell.java:386)
            at org.apache.hadoop.util.GenericOptionsParser.preProcessForWindows(GenericOptionsParser.java:438)
            at org.apache.hadoop.util.GenericOptionsParser.parseGeneralOptions(GenericOptionsParser.java:484)
            at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:170)
            at org.apache.hadoop.util.GenericOptionsParser.<init>(GenericOptionsParser.java:153)
            at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:64)
            at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
            at org.apache.hadoop.fs.FsShell.main(FsShell.java:362)
    16/04/07 16:01:13 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
    Warning: fs.defaultFs is not set when running "ls" command.
    Found 15 items
    -ls: Fatal internal error
    java.lang.NullPointerException
            at java.lang.ProcessBuilder.start(ProcessBuilder.java:1010)
            at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
            at org.apache.hadoop.util.Shell.run(Shell.java:478)
            at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:738)
            at org.apache.hadoop.util.Shell.execCommand(Shell.java:831)
            at org.apache.hadoop.util.Shell.execCommand(Shell.java:814)
            at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1100)
            at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.loadPermissionInfo(RawLocalFileSystem.java:582)
            at org.apache.hadoop.fs.RawLocalFileSystem$DeprecatedRawLocalFileStatus.getOwner(RawLocalFileSystem.java:565)
            at org.apache.hadoop.fs.shell.Ls.adjustColumnWidths(Ls.java:139)
            at org.apache.hadoop.fs.shell.Ls.processPaths(Ls.java:110)
            at org.apache.hadoop.fs.shell.Command.recursePath(Command.java:373)
            at org.apache.hadoop.fs.shell.Ls.processPathArgument(Ls.java:98)
            at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:271)
            at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:255)
            at org.apache.hadoop.fs.shell.FsCommand.processRawArguments(FsCommand.java:118)
            at org.apache.hadoop.fs.shell.Command.run(Command.java:165)
            at org.apache.hadoop.fs.FsShell.run(FsShell.java:305)
            at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
            at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
            at org.apache.hadoop.fs.FsShell.main(FsShell.java:362)

对于上面的问题，我应该进一步尝试修复这个问题，这样我就可以重用我现有的客户机.sh脚本，或者只是将它们转换为.bat？

Answer 1

问题是，cygwin需要返回窗口路径，而不是cygwin路径。此外，需要在路径中安装winutils.exe，正如所描述的这里

只需修复脚本以返回实际的win路径，并关闭一些在cygwin下不运行的命令

#!/bin/bash
# fix $HADOOP_HOME/bin/hdfs
sed -i -e "s/bin=/#bin=/g" $HADOOP_HOME/bin/hdfs
sed -i -e "s#DEFAULT_LIBEXEC_DIR=\"\$bin\"/../libexec#DEFAULT_LIBEXEC_DIR=\"\$HADOOP_HOME\\\libexec\"#g" $HADOOP_HOME/bin/hdfs
sed -i "/export CLASSPATH=$CLASSPATH/i CLASSPATH=\`cygpath -p -w \"\$CLASSPATH\"\`" $HADOOP_HOME/bin/hdfs

# fix $HADOOP_HOME/libexec/hdfs-config.sh
sed -i -e "s/bin=/#bin=/g" $HADOOP_HOME/libexec/hdfs-config.sh
sed -i -e "s#DEFAULT_LIBEXEC_DIR=\"\$bin\"/../libexec#DEFAULT_LIBEXEC_DIR=\"\$HADOOP_HOME\\\libexec\"#g" $HADOOP_HOME/libexec/hdfs-config.sh

# fix $HADOOP_HOME/libexec/hadoop-config.sh
sed -i "/HADOOP_DEFAULT_PREFIX=/a HADOOP_PREFIX=" $HADOOP_HOME/libexec/hadoop-config.sh
sed -i "/export HADOOP_PREFIX/i HADOOP_PREFIX=\`cygpath -p -w \"\$HADOOP_PREFIX\"\`" $HADOOP_HOME/libexec/hadoop-config.sh

# fix $HADOOP_HOME/bin/hadoop 
sed -i -e "s/bin=/#bin=/g" $HADOOP_HOME/bin/hadoop 
sed -i -e "s#DEFAULT_LIBEXEC_DIR=\"\$bin\"/../libexec#DEFAULT_LIBEXEC_DIR=\"\$HADOOP_HOME\\\libexec\"#g" $HADOOP_HOME/bin/hadoop 
sed -i "/export CLASSPATH=$CLASSPATH/i CLASSPATH=\`cygpath -p -w \"\$CLASSPATH\"\`" $HADOOP_HOME/bin/hadoop