如何在tensorflow_transform中获得apply_vocabulary之前的词汇量？

Question

也在https://github.com/tensorflow/transform/issues/261上发布了这个问题

我在TFX中使用tft，需要将字符串列表类标签转换为preprocesing_fn中的多个热点指示器。实质上：

vocab = tft.vocabulary(inputs['label'])
outputs['label'] = tf.cast(
    tf.sparse.to_indicator(
       tft.apply_vocabulary(inputs['label'], vocab),
       vocab_size=VOCAB_SIZE,
    ),
    "int64",
)

我试图从find的结果中获得VOCAB_SIZE，但无法找到满足延迟执行和已知形状的方法。下面最接近的是不会传递保存的模型导出，因为标签的形状是未知的。

def _make_table_initializer(filename_tensor):
    return tf.lookup.TextFileInitializer(
        filename=filename_tensor,
        key_dtype=tf.string,
        key_index=tf.lookup.TextFileIndex.WHOLE_LINE,
        value_dtype=tf.int64,
        value_index=tf.lookup.TextFileIndex.LINE_NUMBER,
    )

def _vocab_size(deferred_vocab_filename_tensor):
    initializer = _make_table_initializer(deferred_vocab_filename_tensor)
    table = tf.lookup.StaticHashTable(initializer, default_value=-1)
    table_size = table.size()
    return table_size

deferred_vocab_and_filename = tft.vocabulary(inputs['label'])
vocab_applied = tft.apply_vocabulary(inputs['label'], deferred_vocab_and_filename)
vocab_size = _vocab_size(deferred_vocab_and_filename)
outputs['label'] = tf.cast(
    tf.sparse.to_indicator(vocab_applied, vocab_size=vocab_size),
    "int64",
)

得到

ValueError: Feature label (Tensor("Identity_3:0", shape=(None, None), dtype=int64)) had invalid shape (None, None) for FixedLenFeature: apart from the batch dimension, all dimensions must have known size [while running 'Analyze/CreateSavedModel[tf_v2_only]/CreateSavedModel']

知道如何做到这一点吗？

Answer 1

根据github中的这个评论，您可以使用tft.experimental.get_vocabulary_size_by_name (链接)来实现相同的功能。