如何使用pandas将句子拆分成句子Id、单词和标签？

Question

我想将我的pandas数据帧转换为可以在NER模型中使用的格式。

我有一个像这样的熊猫数据框架：

Sentence_id语句标签

%1不喜欢新的Windows 8和触摸屏功能。Windows 8

%1不喜欢新的Windows 8和触摸屏功能。触摸屏功能

能不能把它转换成下面的格式？

Sentence_id words标签

1个Did O

1非O

1享受O

1.O

1个新O

1个Windows B

1 8 i

1和O

1个触摸屏B

1个函数i

1.O

标签中的第一个单词应标记为‘B’(开头)，标签中的以下单词应标记为‘I’(内部)。其他单词和标点符号应标记为O(外部)。

Answer 1

这个解决方案有点长。但您可以使用df.iterrows()来完成此操作。

import string

ids = df.Sentence_id.unique().tolist()     ## Assuming name of your dataframe is df
sentences = df.Sentence.unique().tolist()
labels = df.labels.unique().tolist()

def get_label(word, labels):
  if word == labels[0]:
    return 'B'
  elif word in labels and word!= labels[0]:
    return 'I'
  else:
    return 'O'

data = {}
exclude = set(string.punctuation)
for _, row in df.iterrows():
  words = ''.join(ch for ch in row['Sentence'] if ch not in exclude).split()
  puncts = ''.join(ch for ch in row['Sentence'] if ch in exclude).split()
  labels = row['labels'].split()
  for word in words: 
    if word in data:
      if word in labels:
        data[word][1] =  get_label(word, labels)
    else:
      data[word] = [row['Sentence_id'], get_label(word, labels)]
    for punct in puncts:
      data[punct] = [row['Sentence_id'],'O']

## Processing the dictionary to input into dataframe
ids = []
words = []
labels = []
for key, val in data.items():
  words.append(key)
  ids.append(data[key][0])
  labels.append(data[key][1])
new_df = pd.DataFrame({'Sentence_id':ids, 'words':words, 'labels':labels})
new_df

    Sentence_id words   labels
0   1           Did     O
1   1           .       O
2   1           not     O
3   1           enjoy   O
4   1           the     O
5   1           new     O
6   1           Windows B
7   1           8       I
8   1           and     O
9   1       touchscreen B
10  1         functions I