在python中，Word_tokenize在sent_tokenize之后不工作。

Question

我试图使用sent_tokenize和word_tokenize通过数据来标记。

下面是我的虚拟数据

**短信**你好世界，你好我很好，谢谢！

我试图使用下面的代码来标记它

import pandas as pd
from nltk.tokenize import word_tokenize, sent_tokenize
Corpus=pd.read_csv(r"C:\Users\Desktop\NLP\corpus.csv",encoding='utf-8')

Corpus['text']=Corpus['text'].apply(sent_tokenize)
Corpus['text_new']=Corpus['text'].apply(word_tokenize)

但是它给出了下面的错误

Traceback (most recent call last):
  File "C:/Users/gunjit.bedi/Desktop/NLP Project/Topic Classification.py", line 24, in <module>
    Corpus['text_new']=Corpus['text'].apply(word_tokenize)
  File "C:\ProgramData\Anaconda3\lib\site-packages\pandas\core\series.py", line 3192, in apply
    mapped = lib.map_infer(values, f, convert=convert_dtype)
  File "pandas/_libs/src\inference.pyx", line 1472, in pandas._libs.lib.map_infer
  File "C:\ProgramData\Anaconda3\lib\site-packages\nltk\tokenize\__init__.py", line 128, in word_tokenize
    sentences = [text] if preserve_line else sent_tokenize(text, language)
  File "C:\ProgramData\Anaconda3\lib\site-packages\nltk\tokenize\__init__.py", line 95, in sent_tokenize
    return tokenizer.tokenize(text)
  File "C:\ProgramData\Anaconda3\lib\site-packages\nltk\tokenize\punkt.py", line 1241, in tokenize
    return list(self.sentences_from_text(text, realign_boundaries))
  File "C:\ProgramData\Anaconda3\lib\site-packages\nltk\tokenize\punkt.py", line 1291, in sentences_from_text
    return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]
  File "C:\ProgramData\Anaconda3\lib\site-packages\nltk\tokenize\punkt.py", line 1291, in <listcomp>
    return [text[s:e] for s, e in self.span_tokenize(text, realign_boundaries)]
  File "C:\ProgramData\Anaconda3\lib\site-packages\nltk\tokenize\punkt.py", line 1281, in span_tokenize
    for sl in slices:
  File "C:\ProgramData\Anaconda3\lib\site-packages\nltk\tokenize\punkt.py", line 1322, in _realign_boundaries
    for sl1, sl2 in _pair_iter(slices):
  File "C:\ProgramData\Anaconda3\lib\site-packages\nltk\tokenize\punkt.py", line 313, in _pair_iter
    prev = next(it)
  File "C:\ProgramData\Anaconda3\lib\site-packages\nltk\tokenize\punkt.py", line 1295, in _slices_from_text
    for match in self._lang_vars.period_context_re().finditer(text):
TypeError: expected string or bytes-like object

我确实尝试过很多事情，比如如果我评论sent_tokenize，word_tokenize可以工作，但是两者都不能一起工作。

Answer 1

您将得到一个错误，因为nltk.word_tokenize期望输入为string。

当您在文本上应用nltk.sent_tokenize时，它会将其转换为列表。

text = ['Hey. Hello','hello world!! I am akshay','I m fine']

df['text']=df['text'].apply(sent_tokenize)
print(df['text'])

输出：

                           text
0                 [Hey., Hello]
1  [hello world!!, I am akshay]
2                    [I m fine]

试试这个

df['sent']=df['text'].apply(lambda x :sent_tokenize(str(x)))

df['text_new']= [word_tokenize(str(i)) for i in df['sent']]