猪拉丁语词数统计

Question

我正在尝试计算包含以下单词的行数:猪脚本中的'jack', 'hack', 'mat', 'throttle'。我正在使用Cloudera quickstart vm。

输入文件为：

09-jack-17,5:00PM;#slowmotion,Tribune Logic hack: how is life in temrs of money Creative hack

14-June-18,7:15PM;#Indiacalling,Horton-NJ Strategic/Halloween One World at Application Deployment

12-jack-16,jfh:er;#temporary, accomodation, osteoporosis, juxtapose, don't misinterpret this awaiting throttle jack

输出应该是: hack 2 jack 2油门1 mat 0我无法提取这些单词并计算它的计数。我该怎么办？我尝试了inquisitive_mind提供的以下脚本：

A = LOAD 'Input.txt'AS(line: chararray);
SPLIT A INTO M IF line matches'hackathon,N IF line matches'dec', O IF line matches'chicago',P IF line matches'java';
M1 = GROUP M ALL;
M2 = FOR EACH M1 GENERATE COUNT(M);
M3 = FOREACH M2 GENERATE CONCAT('hackathon',(chararray)M2.$0);
N1 = GROUP N ALL;
N2 = FOREACH N1 GENERATE COUNT(N);
N3 = FOREACHN2 GENERATE CONCAT('dec',(chararray)N2.$0);
O1 = GROUP O ALL;
O2 = FOREACH O1 GENERATE COUNT(O);
O3 = FOR EACH O2 GENERATE CONCAT('chicago',(chararray)O2.$0);
P1 = GROUP P ALL;
P2 = FOR EACH P1 GENERATE COUNT(P);
P3 = FOREACH P2 GENERATE CONCAT('java',(chararray)P2.$0);
DUMP M3;
DUMP N3;
DUMP O3;
DUMP P3;

但是当我在mapreduce或本地模式下运行它时，我得到以下错误：

2016-10-11 09:43:44,084 main ERROR org.apache.pig.tools.grunt.Grunt - ERROR 1000:解析时出错。第19行，第0列出现词法错误。遇到:之后："“详细信息位于日志文件: /home/cloudera/pig_1476204218406.log

以下是日志文件：

猪堆栈跟踪

ERROR 1000: Error during parsing. Lexical error at line 19, column 0.  Encountered: <EOF> after : ""

org.apache.pig.tools.pigscript.parser.TokenMgrError: Lexical error at line 19, column 0.  Encountered: <EOF> after : ""
    at org.apache.pig.tools.pigscript.parser.PigScriptParserTokenManager.getNextToken(PigScriptParserTokenManager.java:3326)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.jj_ntk(PigScriptParser.java:1379)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:106)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:198)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:173)
    at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:84)
    at org.apache.pig.Main.run(Main.java:613)
    at org.apache.pig.Main.main(Main.java:158)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Answer 1

如果单词是固定的且数量很少，您可以使用split并计算行数。

A = LOAD 'data.txt' AS (line:chararray);
SPLIT A INTO M IF line matches 'jack', N IF line matches 'hack',O IF line matches 'throttle', P IF line matches 'mat';

M1 = GROUP M ALL;
M2 = FOREACH M1 COUNT(M); 
M3 = FOREACH M2 GENERATE CONCAT('jack ',(chararray)M2.$0); 

N1 = GROUP N ALL;
N2 = FOREACH N1 COUNT(N); 
N3 = FOREACH N2 GENERATE CONCAT('hack ',(chararray)N2.$0);

O1 = GROUP O ALL;
O2 = FOREACH O1 COUNT(O); 
O3 = FOREACH O2 GENERATE CONCAT('throttle ',(chararray)O2.$0);

P1 = GROUP P ALL;
P2 = FOREACH P1 COUNT(P); 
P3 = FOREACH P2 GENERATE CONCAT('mat ',(chararray)P2.$0);

DUMP M3;
DUMP N3;
DUMP O3;
DUMP P3;