如何用nltk库从dataframes中分割句子？

Question

我想要创造袋的文字模型，但与计算相对频率与nltk包。我的数据是用熊猫数据建立的。

这是我的数据：

text    title   authors label
0   On Saturday, September 17 at 8:30 pm EST, an e...   Another Terrorist Attack in NYC…Why Are we STI...   ['View All Posts', 'Leonora Cravotta']  Real
1   Story highlights "This, though, is certain: to...   Hillary Clinton on police shootings: 'too many...   ['Mj Lee', 'Cnn National Politics Reporter']    Real
2   Critical Counties is a CNN series exploring 11...   Critical counties: Wake County, NC, could put ...   ['Joyce Tseng', 'Eli Watkins']  Real
3   McCain Criticized Trump for Arpaio’s Pardon… S...   NFL Superstar Unleashes 4 Word Bombshell on Re...   []  Real
4   Story highlights Obams reaffirms US commitment...   Obama in NYC: 'We all have a role to play' in ...   ['Kevin Liptak', 'Cnn White House Producer']    Real
5   Obama weighs in on the debate\n\nPresident Bar...   Obama weighs in on the debate   ['Brianna Ehley', 'Jack Shafer']    Real

我试着把它转换成字符串

import nltk 
import numpy as np
import random
import bs4 as bs
import re

data = df.astype(str)
data

但是，当我试图标记这个单词时，它有如下错误

corpus = nltk.sent_tokenize(data['text'])

TypeError: expected string or bytes-like object

但似乎行不通：(有人知道如何标记“文本”列中的每一行句子吗？)

Answer 1

nltk.tokenize()要求输入为字符串，您将得到错误，因为您正在直接传递一个pandas.Series对象：

试着用这个词来标记：

data['Corpus'] = df.text.apply(lambda x: nltk.word_tokenize(x))

对于sent_tokenize修改：

data['Sent'] = df.text.apply(lambda x: nltk.sent_tokenize(x))

如果您还想去掉标点符号：

data['no_punc'] = df.text.apply(lambda x: nltk.RegexpTokenizer(r'\w+').tokenize(x))