Hadoop TotalOrderPartitioner

Question

我正在尝试使用hadoop中的total order partioner，代码如下：

  job.setNumReduceTasks(4);
  Path partitionFile = new Path(args[1]);
  InputSampler.Sampler sampler = new InputSampler.RandomSampler(0.1,3,1)
  TotalOrderPartitioner.setPartitionFile(job.getConfiguration(),partitionFile);
  InputSampler.writePartitionFile(job, sampler);
  job.setPartitionerClass(TotalOrderPartitioner.class); 
  FileInputFormat.addInputPath(job, new Path(otherArgs[0));
  FileOutputFormat.setOutputPath(job, new Path(otherArgs[2]));

Exception running child : java.lang.IllegalArgumentException: Can't read partitions file
    at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:116)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:677)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:746)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:339)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:162)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:157)
Caused by: java.io.FileNotFoundException: File file:/tmp/hadoop-sarang/nm-local-dir/usercache/sarang/appcache/application_1417956066584_0001/container_1417956066584_0001_01_000005/_partition.lst does not exist
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:520)
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
    at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1749)
    at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1773)
    at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.readPartitions(TotalOrderPartitioner.java:301)
    at org.apache.hadoop.mapreduce.lib.partition.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:88)
    ... 10 more

Answer 1

当我使用hadoop mapreduce并且mapreduce服务没有安装和启动时，我也发现了这个错误。安装mapreduce并启动它后，异常消失了。