fs.hdfs.hadoopconf在flink-flink.flink中的用途

Question

新手到弗林克。

我能够在远程hdfs集群中的文件上运行示例wordcount.jar，而无需在flink中声明fs.hdfs.hadoopconf变量。

所以想知道上面提到的变量的目的到底是什么。

声明它是否改变了运行示例jar的方式？

指挥：

flink-cluster.vm ~]$ /opt/flink/bin/flink run  /opt/flink/examples/batch/WordCount.jar --input hdfs://hadoop-master:9000/tmp/test-events

输出：

.......
07/13/2016 00:50:13 Job execution switched to status FINISHED.
(foo,1)
.....
(bar,1)
(one,1)

设置：

• HDFS上的远程hdfs集群://hadoop-master.vm. on :9000
• 在Flink上运行的flink集群. on

谢谢

更新：

正如Serhiy所指出的，在conf中声明了fs.hdfs.hadoopconf，但是在运行带有更新参数的作业时，hdfs:///tmp/test-events.1468374669125得到了以下错误

flink-conf.yaml

# You can also directly specify the paths to hdfs-default.xml and hdfs-site.xml
# via keys 'fs.hdfs.hdfsdefault' and 'fs.hdfs.hdfssite'.
#
fs.hdfs.hadoopconf: hdfs://hadoop-master:9000/
fs.hdfs.hdfsdefault :  hdfs://hadoop-master:9000/

指挥：

flink-cluster.vm ~]$ /opt/flink/bin/flink run  /opt/flink/examples/batch/WordCount.jar --input hdfs:///tmp/test-events

产出：

Caused by: org.apache.flink.runtime.JobException: Creating the input splits caused an error: The given HDFS file URI (hdfs:///tmp/test-events.1468374669125) did not describe the HDFS NameNode. The attempt to use a default HDFS configuration, as specified in the 'fs.hdfs.hdfsdefault' or 'fs.hdfs.hdfssite' config parameter failed due to the following problem: Either no default file system was registered, or the provided configuration contains no valid authority component (fs.default.name or fs.defaultFS) describing the (hdfs namenode) host and port.
    at org.apache.flink.runtime.executiongraph.ExecutionJobVertex.<init>(ExecutionJobVertex.java:172)
    at org.apache.flink.runtime.executiongraph.ExecutionGraph.attachJobGraph(ExecutionGraph.java:679)
    at org.apache.flink.runtime.jobmanager.JobManager.org$apache$flink$runtime$jobmanager$JobManager$$submitJob(JobManager.scala:1026)
    ... 19 more

Answer 1

来自文档

fs.hdfs.hadoopconf：通往Hadoop的配置目录(可选值)的绝对路径。指定此值允许程序使用短URI (hdfs:///path/to/files，而不包括文件URI中的NameNode地址和端口)引用HDFS文件。如果没有此选项，则可以访问HDFS文件，但需要像hdfs://address:port/path/to/files这样的完全限定URI。此选项还会导致文件编写器获取HDFS块大小和复制因子的默认值。Flink将在指定的目录中查找“core-site.xml”和“hdfs-site.xml”文件。