谷歌Colab中的SentencePiece

Question

我想在Google Colab项目中使用来自https://github.com/google/sentencepiece的句子指示器，在该项目中我正在训练一个OpenNMT模型。我对如何在Google Colab中设置判决器二进制文件感到有点困惑。我需要用cmake构建吗？

当我尝试使用pip install sentencepiece进行安装，并试图在脚本的“转换”中包含句子摘要时，我得到了以下错误

运行此脚本(与OpenNMT翻译教程中的脚本匹配) !onmt_build_vocab -config en-sp.yaml -n_sample -1

我得到了：

Traceback (most recent call last):
  File "/usr/local/bin/onmt_build_vocab", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/onmt/bin/build_vocab.py", line 63, in main
    build_vocab_main(opts)
  File "/usr/local/lib/python3.7/dist-packages/onmt/bin/build_vocab.py", line 32, in build_vocab_main
    transforms = make_transforms(opts, transforms_cls, fields)
  File "/usr/local/lib/python3.7/dist-packages/onmt/transforms/transform.py", line 176, in make_transforms
    transform_obj.warm_up(vocabs)
  File "/usr/local/lib/python3.7/dist-packages/onmt/transforms/tokenize.py", line 110, in warm_up
    load_src_model.Load(self.src_subword_model)
  File "/usr/local/lib/python3.7/dist-packages/sentencepiece/__init__.py", line 367, in Load
    return self.LoadFromFile(model_file)
  File "/usr/local/lib/python3.7/dist-packages/sentencepiece/__init__.py", line 171, in LoadFromFile
    return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
TypeError: not a string

下面是我的脚本是如何写的。我不确定这个字符串是从哪里来的。

## Where the samples will be written
save_data: en-sp/run/example

## Where the vocab(s) will be written
src_vocab: en-sp/run/example.vocab.src
tgt_vocab: en-sp/run/example.vocab.tgt

## Where the model will be saved
save_model: drive/MyDrive/Europarl/model/model

# Prevent overwriting existing files in the folder
overwrite: False

# Corpus opts:
data:
    europarl:
        path_src: train_europarl-v7.es-en.es
        path_tgt: train_europarl-v7.es-en.en
        transforms: [sentencepiece, filtertoolong]
        weight: 1

    valid:
        path_src: dev_europarl-v7.es-en.es
        path_tgt: dev_europarl-v7.es-en.en
        transforms: [sentencepiece]

skip_empty_level: silent

world_size: 1
gpu_ranks: [0]
...

编辑:所以我继续用谷歌搜索这个问题，找到了一个谷歌colab项目，它使用cmake here https://colab.research.google.com/github/mymusise/gpt2-quickly/blob/main/examples/gpt2_quickly.ipynb#scrollTo=dDAup5dxDXZW构建了句子摘录。然而，即使在使用cmake构建之后，我仍然会遇到这个问题。

Answer 1

为了解决这个问题，我必须过滤和标记化我的数据集，然后使用句子表进行训练。我使用了这个有用的资源中的脚本：https://github.com/ymoslem/MT-Preparation来做所有的事情，现在我的模型正在训练！