如何用MapReduce的第二个选项卡拆分单词？

Question

我正在做一些网络数据的MapReduces。(我是MapReduce新手，所以想想经典的WordCount类型的东西吧。)输入文件如下，数字后面跟着一个选项卡：

3 2 2 4 2 2 2 3

虽然我知道如何获得数字的经典“字数”，但我真正想做的是对数字进行成对的计算，所以上面的数字将被映射者解读为'3 2'，'2 2'，'2‘’，等等。我该怎么做？我想所有需要做的就是调整StringTokenizer，使其通过第二个选项卡或其他什么来分割单词，但我该如何做呢？这可能吗？

下面是我正在使用的Java代码，到目前为止，这只是MapReduce中的经典MapReduce示例：

public static class TokenizerMapper
   extends Mapper<Object, Text, Text, IntWritable>{

private final static IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(Object key, Text value, Context context
                ) throws IOException, InterruptedException {
  StringTokenizer itr = new StringTokenizer(value.toString());
  while (itr.hasMoreTokens()) {
    word.set(itr.nextToken());
    context.write(word, one);
  }
}
}

Answer 1

谢谢你的帮助！这是我想出的解决方案(在添加了一些前导零以帮助格式化之后)：

 public class WordCount {

   public static class TokenizerMapper
        extends Mapper<Object, Text, Text, IntWritable>{

     private final static IntWritable one = new IntWritable(1);
     private Text word = new Text();

     public void map(Object key, Text value, Context context
                ) throws IOException, InterruptedException {
         String data = value.toString();
         for (int i = 0; i < (data.length() / 3) - 1; i++) {
             String pair = data.substring(i*3, (i*3)+5);
             context.write(new Text(pair), one);
         }
      }
   }

Answer 2

您可以轻松地修改WordCount，使其具有预期的行为。

public static class TokenizerMapper 
   extends Mapper<Object, Text, Text, IntWritable>{

private final static IntWritable one = new IntWritable(1);
private Text word = new Text();

public void map(Object key, Text value, Context context
                ) throws IOException, InterruptedException {
  StringTokenizer itr = new StringTokenizer(value.toString());
  while (itr.hasMoreTokens()) {
    word.set(itr.nextToken());
    String myString = word.toString();
    String [] numbers = myString.split("\t"); // split by tab
    if (numbers.length> 2)
    {
        // you need at least two numbers to make one pair
        int first = Integer.parseInt(numbers[0]);
        int second;
        for (int i=1; i < numbers.length; ++i)
        {
           second = Integer.parseInt(numbers[i]);
           Text keynew = new Text(first+"\t"+second);
           context.write(keynew, one);
           // your second will be the first in the next loop iteration
           first = second;
        }
    }
  }
}
}

Answer 3

试试这个：

String data = "0\t0\t1\t2\t4\t5\t3\t4\t6\t7";
String[] array = data.split("(?<=\\G\\w{1,3}\t\\w{1,3})\t");

    for(String s : array){
        System.out.println(s);
    }

其中{1,3}是数字中数字数的范围。

输出：

 0  0

 1  2

 4  5

 3  4

 6  7

为了你的密码，

String[] pairsArray = value.toString().split("(?<=\\G\\w{1,3}\t\\w{1,3})\t");
for (String pair : pairsArray) {
     context.write(new Text(pair), one);
}