从简单的java程序调用Giraph作业

Question

我是Giraph和Hadoop Yarn的新手。通过Giraph的quick start，我可以从命令行运行jar build from source的示例作业。

我想从简单的java程序运行作业。这个问题的灵感来自于之前类似的MapReduce job question。寻找java依赖项的类似答案，这将是需要的。

我有纱线设置在本地-需要有一种方式，从java程序馈送作业。

很明显：https://giraph.apache.org/apidocs/org/apache/giraph/job/GiraphJob.html肯定有办法做到这一点--但我发现很难找到Yarn的例子。

Answer 1

在GiraphRunner的源代码中找到了这样做的方法：

@Test
public void testPageRank() throws IOException, ClassNotFoundException, InterruptedException {

    GiraphConfiguration giraphConf = new GiraphConfiguration(getConf());
    giraphConf.setWorkerConfiguration(1,1,100);
    GiraphConstants.SPLIT_MASTER_WORKER.set(giraphConf, false);

    giraphConf.setVertexInputFormatClass(JsonLongDoubleFloatDoubleVertexInputFormat.class);
    GiraphFileInputFormat.setVertexInputPath(giraphConf,
                                             new Path("/input/tiny-graph.txt"));
    giraphConf.setVertexOutputFormatClass(IdWithValueTextOutputFormat.class);

    giraphConf.setComputationClass(PageRankComputation.class);

    GiraphJob giraphJob = new GiraphJob(giraphConf, "page-rank");       

    FileOutputFormat.setOutputPath(giraphJob.getInternalJob(),
                                   new Path("/output/page-rank2"));
    giraphJob.run(true);
}

private Configuration getConf() {
    Configuration conf = new Configuration();
    conf.set("fs.defaultFS", "hdfs://localhost:9000");

    conf.set("yarn.resourcemanager.address", "localhost:8032");
    conf.set("yarn.resourcemanager.hostname", "localhost");

    // framework is now "yarn", should be defined like this in mapred-site.xm
    conf.set("mapreduce.framework.name", "yarn");
    return conf;
}