Python词干分析(使用pandas数据帧)

Question

我创建了一个数据帧，其中的句子需要进行词干处理。我想使用雪球分类器来获得我的分类算法的更高精度。我如何才能做到这一点？

import pandas as pd
from nltk.stem.snowball import SnowballStemmer

# Use English stemmer.
stemmer = SnowballStemmer("english")

# Sentences to be stemmed.
data = ["programmers program with programming languages", "my code is working so there must be a bug in the interpreter"] 
    
# Create the Pandas dataFrame.
df = pd.DataFrame(data, columns = ['unstemmed']) 

# Split the sentences to lists of words.
df['unstemmed'] = df['unstemmed'].str.split()

# Make sure we see the full column.
pd.set_option('display.max_colwidth', -1)

# Print dataframe.
df 

+----+---------------------------------------------------------------+
|    | unstemmed                                                     |
|----+---------------------------------------------------------------|
|  0 | ['programmers', 'program', 'with', 'programming', 'languages']|
|  1 | ['my', 'code', 'is', 'working', 'so', 'there', 'must',        |  
|    |  'be', 'a', 'bug', 'in', 'the', 'interpreter']                |
+----+---------------------------------------------------------------+

Answer 1

您必须对每个单词应用词干分析，并将其存储到“词干”列中。

df['stemmed'] = df['unstemmed'].apply(lambda x: [stemmer.stem(y) for y in x]) # Stem every word.
df = df.drop(columns=['unstemmed']) # Get rid of the unstemmed column.
df # Print dataframe.

+----+--------------------------------------------------------------+
|    | stemmed                                                      |
|----+--------------------------------------------------------------|
|  0 | ['program', 'program', 'with', 'program', 'languag']         |
|  1 | ['my', 'code', 'is', 'work', 'so', 'there', 'must',          |   
|    |  'be', 'a', 'bug', 'in', 'the', 'interpret']                 |
+----+--------------------------------------------------------------+