规范字符串列表Python

Question

例如，我有一个非常大的字符串列表，列表中的每个单词都是未规范化的：

word_list = ["Alzheimer", "Alzheimer's", "Alzheimer.", "Alzheimer?","Cognition.", "Cognition's", "Cognitions", "Cognition"] # and the list goes on

正如你所看到的，列表中有许多相同的术语，但其中有些包含不必要的词句(例如:点、单撇号)，我如何使所有单词规范化(例如：“Alzheimers。”->“老年痴呆症者”，“认知的”“->”认知“)？

提前谢谢你！

我希望有一个函数可以过滤掉不必要的单个标点符号，我尝试了以下函数，但效果不佳：

def word_normalizer(word): # Remove unnecessary single puntuations and turn all words into lower case
    puntuations = ["'", '"', ";", ":", ",", ".", "&", "(", ")"]
    new_word =""
    for punc in puntuations:
        if punc in word:
            new_word = word.strip(punc)
            
        return new_word.lower()

Answer 1

您可以尝试使用此函数，我尝试过这样做，它删除所有特殊字符(所有非字母/数字)和标点符号，并将语句取为小写字母。

import re
def word_normalizer(word): 
    word = re.sub('[^A-Za-z0-9]+', '', word)
    return word.lower()

Answer 2

标准字符串模块有一个有用的标点符号值(它可能适合你的目的，也可能不适合你)。

您可以方便地使用re模块来处理替换。

下面的代码删除了尾随's‘，这在所有情况下都可能是不可取的--也就是说，仅仅因为一个单词以's’结尾并不一定意味着它是复数

有些替换会导致重复，所以使用一套。

import string
import re

punc = re.compile(f'[{string.punctuation}]')
word_list = ["Alzheimer", "Alzheimer's", "Alzheimer.", "Alzheimer?","Cognition.", "Cognition's", "Cognitions", "Cognition"]
new_word_set = {punc.sub('', word).rstrip('s') for word in word_list}

print(new_word_set)

输出：

{'Alzheimer', 'Cognition'}

Answer 3

您可以使用功能技术非常简洁地完成这一任务。

    from functools import reduce

  punctuation = ["'", '"', ";", ":", ",", ".", "&", "(", ")"]
  words =  ["Alzheimer", "Alzheimer's", "Alzheimer.", "Alzheimer?","Cognition.", "Cognition's", "Cognitions", "Cognition"]

  # remove a character from a string
  def strip_punc(word: str, character_to_strip: str) -> str:
    return word.replace(character_to_strip, "")

  # run the strip_punc function for each item in the punctuation list
  def clean_word(word: str): list(str) -> list(str):
    return reduce(strip_punc, punctuation, word)

  # run the clean_word function on each word in the word list
  # use set to remove dupes
  return set(map(clean_word, words))

您可以使用lambda函数使其更加简洁。

  from functools import reduce

  punctuation = ["'", '"', ";", ":", ",", ".", "&", "(", ")"]
  words =  ["Alzheimer", "Alzheimer's", "Alzheimer.", "Alzheimer?","Cognition.", "Cognition's", "Cognitions", "Cognition"]

  return set(
    map(
     lambda p: reduce(lambda p, w: p.replace(w, ""), punctuation, p), words
    )
  )