使用撇号解除字符串的标记后，删除空格

Question

我想通过regex或detokenizing来删除单词中的空格，比如can't或won't

from nltk.tokenize import WordPunctTokenizer
tok = WordPunctTokenizer()
detok = MosesDetokenizer()

pattern= "[^\w ]+ "
text= "i can ' t use this cause they won ' t fit"
string= re.sub(pattern, '', text)
tk = tok.tokenize(string)
output= detok.detokenize(tk, return_str = True)
print(output)

 "i can 't use this cause they won' t fit"

任何关于如何删除' can‘和'won’之后的空格的想法，这样我就可以拥有can't和When‘t了。当我使用output = (' '.join(tk)).strip()去标记化时，我得到了两个空格，一个在撇号前后。示例i can ' t use this cause they won ' t fit

Answer 1

@BenT我不能说关于正则表达式，但是在你的输出中，你可以应用以下操作：

output = "i can 't use this cause they won' t fit"
output = "'".join(output.split(" '"))
output = "'".join(output.split("' "))
print(output)
"i can't use this cause they won't fit"

这里也有一行解决方案：

output = output.replace("' ", "'").replace(" '", "'")
print(output)
"i can't use this cause they won't fit"

Answer 2

我认为你可以简单地做一些事情，比如：

output = "i can 't use this cause they won' t fit"
output = output.replace(" '", "")
print output
"i can't use this cause they won't fit"