定格词典

Question

我需要做LIWC(语言查询和单词计数)，我正在使用quanteda/quanteda.dictionary。我需要“加载”自定义字典:我将我的单词列表保存为单独的.txt文件和一个“加载”直读行(例如，只有一个字典)：

autonomy = readLines("Dictionary/autonomy.txt", encoding = "UTF-8")

EODic<-quanteda::dictionary(list(autonomy=autonomy),encoding = "auto")

这是我正在试用的一段文字。

txt <- c("12th Battalion Productions is producing a fully holographic feature length production. Presenting a 3D audio-visual projection without a single cast member present, to give the illusion of live stage performance.")

然后我运行它：

liwcalike(txt, EODic, what = "word")

得到这个错误：

Error in stri_replace_all_charclass(value, "\\p{Z}", concatenator) : 


invalid UTF-8 byte sequence detected; perhaps you should try calling stri_enc_toutf8()

显然，问题在于我的txt文件。我有相当多的字典，我宁愿把它们作为文件载入。

如何纠正此错误？在读行中指定编码似乎没有帮助。

这是文件https://drive.google.com/file/d/12plgfJdMawmqTkcLWxD1BfWdaeHuPTXV/view?usp=sharing

更新:在Mac上解决这一问题的最简单方法是在Word中打开.txt文件，而不是TextEdit。Word给出了与默认TextEdit不同的编码选项！

Answer 1

好的，问题不是编码问题，因为您所链接的文件中的所有内容都可以完全用较低的128个字符ASCII进行编码。问题是空行造成的空白。还有一些前导空间需要移除。这很容易使用一些子设置和一些清理操作。

library("quanteda")
## Package version: 1.3.14

autonomy <- readLines("~/Downloads/risktaking.txt", encoding = "UTF-8")
head(autonomy, 15)
##  [1] "adventuresome"  " adventurous"   " audacious"     " bet"          
##  [5] " bold"          " bold-spirited" " brash"         " brave"        
##  [9] " chance"        " chancy"        " courageous"    " danger"       
## [13] ""               "dangerous"      " dare"

# strip leading or trailing whitespace
autonomy <- stringi::stri_trim_both(autonomy)
# get rid of empties
autonomy <- autonomy[!autonomy == ""]

现在您可以创建字典并应用quanteda.dictionaries::liwcalike()函数。

# now define the quanteda dictionary
EODic <- dictionary(list(autonomy = autonomy))

txt <- c("12th Battalion Productions is producing a fully holographic feature length production. Presenting a 3D audio-visual projection without a single cast member present, to give the illusion of live stage performance.")

library("quanteda.dictionaries")
liwcalike(txt, dictionary = EODic)
##   docname Segment WC  WPS Sixltr Dic autonomy AllPunc Period Comma Colon
## 1   text1       1 35 15.5  34.29   0        0   11.43   5.71  2.86     0
##   SemiC QMark Exclam Dash Quote Apostro Parenth OtherP
## 1     0     0      0 2.86     0       0       0   8.57