如何使用tensorflow标记文本？

Question

我试图使用以下代码来向量化一个句子：

from tensorflow.keras.layers import TextVectorization

text_vectorization_layer =  TextVectorization(max_tokens=10000,
                                              ngrams=5,
                                              standardize='lower_and_strip_punctuation',
                                              output_mode='int',
                                              output_sequence_length = 15
                                              )

text_vectorization_layer(['BlackBerry Limited is a Canadian software'])

但是，它会抱怨以下错误：

AttributeError: 'NoneType' object has no attribute 'ndims'

Answer 1

您必须首先使用TextVectorization方法或通过将词汇表数组传递给该层的vocabulary参数来计算该层的词汇表。下面是一个有用的例子：

import tensorflow as tf

text_vectorization_layer =  tf.keras.layers.TextVectorization(max_tokens=10000,
                                              ngrams=5,
                                              standardize='lower_and_strip_punctuation',
                                              output_mode='int',
                                              output_sequence_length = 15
                                              )

text_vectorization_layer.adapt(['BlackBerry Limited is a Canadian software'])
print(text_vectorization_layer(['BlackBerry Limited is a Canadian software']))

tf.Tensor([[18  7 11 21 13  2 17  6 10 20 12 16  5  9 19]], shape=(1, 15), dtype=int64)

字符串在内部被标记。另外，检查文档。