Java - matcher重读单词

Question

我正在尝试使用java为Delphi创建一个词法分析器。下面是示例代码：

String[] keywords={"array","as","asm","begin","case","class","const","constructor","destructor","dispinterface","div","do","downto","else","end","except","exports","file","finalization","finally","for","function","goto","if","implementation","inherited","initialization","inline","interface","is","label","library","mod","nil","object","of","out","packed","procedure","program","property","raise","record","repeat","resourcestring","set","shl","shr","string","then","threadvar","to","try","type","unit","until","uses","var","while","with"};
String[] relation={"=","<>","<",">","<=",">="};
String[] logical={"and","not","or","xor"};
Matcher matcher = null;
for(int i=0;i<keywords.length;i++){
   matcher=Pattern.compile(keywords[i]).matcher(line); 
   if(matcher.find()){
    System.out.println("Keyword"+"\t\t"+matcher.group());
   }
}
for(int i1=0;i1<logical.length;i1++){
   matcher=Pattern.compile(logical[i1]).matcher(line); 
   if(matcher.find()){
     System.out.println("logic_op"+"\t\t"+matcher.group());
   }
}    
for(int i2=0;i2<relation.length;i2++){
   matcher=Pattern.compile(relation[i2]).matcher(line); 
   if(matcher.find()){
     System.out.println("relational_op"+"\t\t"+matcher.group());
   }
}

所以，当我运行这个程序时，它可以工作，但是它需要重新读取程序认为是2令牌的某些单词--例如：record是一个关键字，但是重新读取它以查找来自的令牌逻辑运算符的单词或。我怎样才能抵消重读单词的影响？谢谢!

Answer 1

正如在answer by EvanM中提到的，您需要在关键字之前和之后添加一个\b单词边界匹配器，以防止单词中的子字符串匹配。

为了获得更好的性能，您还应该使用|逻辑regex操作符来匹配多个值中的一个，而不是创建多个匹配器，因此您只需要扫描line一次，并且只需要编译一个正则表达式。

您甚至可以在一个正则表达式中组合您要寻找的3种不同类型的令牌，并使用捕获组来区分它们，因此您只需要总共扫描一次line。

如下所示：

String regex = "\\b(array|as|asm|begin|case|class|const|constructor|destructor|dispinterface|div|do|downto|else|end|except|exports|file|finalization|finally|for|function|goto|if|implementation|inherited|initialization|inline|interface|is|label|library|mod|nil|object|of|out|packed|procedure|program|property|raise|record|repeat|resourcestring|set|shl|shr|string|then|threadvar|to|try|type|unit|until|uses|var|while|with)\\b" +
              "|(=|<[>=]?|>=?)" +
              "|\\b(and|not|or|xor)\\b";
for (Matcher m = Pattern.compile(regex).matcher(line); m.find(); ) {
    if (m.start(1) != -1) {
        System.out.println("Keyword\t\t" + m.group(1));
    } else if (m.start(2) != -1) {
        System.out.println("logic_op\t\t" + m.group(2));
    } else {
        System.out.println("relational_op\t\t" + m.group(3));
    }
}

您甚至可以通过将关键字与普通前缀相结合来进一步优化它，例如as|asm可以变成asm?，也就是as后面跟着m。将使关键字列表的可读性降低，但性能会更好。

在上面的代码中，我这样做是为了逻辑操作，说明如何修复原始代码中的匹配错误，其中line中的line将以=、>、>=的形式显示3次，这个问题类似于问题中提出的子关键字问题。

Answer 2

将\b添加到正则表达式中，以表示单词之间的间隔。所以：

Pattern.compile("\\b" + keywords[i] + "\\b")

将确保你的单词两边的字符不是字母。

这样，“记录”只能与“记录”相匹配，而不是“或”。