python中的文本数据预处理

Question

我正在提取积极的，消极的和中性的关键字在python.There是10,000条评论在我的评论remarks.txt文件(编码的UTF-8).I要导入文本文件，读取个别行的评论和提取单词(标记化)从评论中提到的列c2，并存储在下一个相邻的列。我用Python.I编写了一个调用get_keywords函数的小程序，我创建了get_keywords()函数，但遇到了将数据帧的每一行作为参数传递&使用迭代调用它并将其存储在相邻列中的问题。

代码没有为df数据帧中的所有已处理字提供预期的列“标记”。

    import nltk
    import pandas as pd
    import re
    import string
    from nltk import sent_tokenize, word_tokenize
    from nltk.corpus import stopwords
    from nltk.stem.porter import PorterStemmer
    remarks = pd.read_csv('/Users/ZKDN0YU/Desktop/comments/New 
    comments/ccomments.txt')
    df = pd.DataFrame(remarks, columns= ['c2'])
    df.head(50)
    df.tail(50)

    filename = 'ccomments.txt'
    file = open(filename, 'rt', encoding="utf-8")
    text = file.read()
    file.close()

    def get_keywords(row):     
    # split into tokens by white space
      tokens = text.split(str(row))
    # prepare regex for char filtering
      re_punc = re.compile('[%s]' % re.escape(string.punctuation))
    # remove punctuation from each word
      tokens = [re_punc.sub('', w) for w in tokens]
    # remove remaining tokens that are not alphabetic
      tokens = [word for word in tokens if word.isalpha()]
    # filter out stop words
      stop_words = set(stopwords.words('english'))
      tokens = [w for w in tokens if not w in stop_words]
    # stemming of words
      porter = PorterStemmer()
      stemmed = [porter.stem(word) for word in tokens]
    # filter out short tokens
      tokens = [word for word in tokens if len(word) > 1]
      return tokens
      df['tokens'] = df.c2.apply(lambda row: get_keywords(row['c2']), 
       axis=1)
      for index, row in df.iterrows():
      print(index, row['c2'],"tokens : {}".format(row['tokens']))

预期输出：-包含列1)索引、2)c2(注释)&3)数据帧所有行的标记化单词的Comments_modified文件，这些数据帧具有10,000条注释。

Answer 1

假设您的文本文件ccomments.txt没有任何标题(即数据从第一行本身开始)，并且每行只有一个列数据(即文本文件只有注释)，下面的代码将返回一个单词列表。

import nltk
import pandas as pd
import re
import string
from nltk import sent_tokenize, word_tokenize
from nltk.corpus import stopwords
from nltk.stem.porter import PorterStemmer


def get_keywords(row):     
    # split into tokens by white space
      tokens = row.split()
    # prepare regex for char filtering
      re_punc = re.compile('[%s]' % re.escape(string.punctuation))
    # remove punctuation from each word
      tokens = [re_punc.sub('', w) for w in tokens]
    # remove remaining tokens that are not alphabetic
      tokens = [word for word in tokens if word.isalpha()]
    # filter out stop words
      stop_words = set(stopwords.words('english'))
      tokens = [w for w in tokens if w not in stop_words]
    # stemming of words
      porter = PorterStemmer()
      stemmed = [porter.stem(word) for word in tokens]
    # filter out short tokens
      tokens = [word for word in tokens if len(word) > 1]
      return tokens


df = pd.read_csv('ccomments.txt',header=None,names = ['c2'])                      
df['tokens'] = df.c2.apply(lambda row: get_keywords(row))
for index, row in df.iterrows():
    print(index, row['c2'],"tokens : {}".format(row['tokens']))