在FiftyOne应用程序中序列化/反序列化图像掩码

Question

我想使用FiftyOne应用程序来评估实例分段预测。我使用可下载的快速启动数据集作为开始，并为地面真相和预测数据创建了自己的samples.json文件，这些数据可以由Dataset.from_dir加载。本教程中的json文件只包含边界框，但不包含掩码。在文档中，我只找到设置为None的掩码，但我需要它们。

{
  "samples": [
    {
      "filepath": "some_image.png",
      "tags": [
        "validation"
      ],
      "metadata": null,
      "uniqueness": 0.998,
      "ground_truth": {
        "_cls": "Detections",
        "detections": [
          {
            "_id": {
              "$oid": "efd81c917c9a49dc9f8aca3f"
            },
            "_cls": "Detection",
            "label": "1",
            "bounding_box": [
              0.62609375,
              0.27044921875,
              0.030078124999999956,
              0.009550781250000001
            ],
            "mask": ???,
          }
        ]
      },
      "predictions": {
        "_cls": "Detections",
        "detections": [
          {
            "_id": {
              "$oid": "6db44c71c1414b8989c92255"
            },
            "_cls": "Detection",
            "label": "1",
            "bounding_box": [
              0.3303889036178589,
              0.4432219862937927,
              0.07914596796035767,
              0.02226179838180542
            ],
            "mask": ???,
            "confidence": 0.999962329864502
          },
        ]
      }
    }
  ]
}

我遇到的问题是如何为分段掩码创建mask属性？它是一个numpy数组，默认情况下不能序列化。Dataset.from_dir使用core.utils.deserialize_numpy_array函数加载数据集，因此我尝试使用serialize_numpy_array保存数据集，但没有成功。

那么，将掩码写入可反序列化的json文件的最佳方法是什么？谢谢!

Answer 1

Dataset.from_dir()语法通常用于定义良好的数据集类型，如以COCODetectionDataset格式存储在磁盘上的数据集。

在您的情况下，当加载与相关数据集类型不直接对应的自定义数据集时，建议的方法是编写简单的Python循环并一次构造一个样本。

在这种情况下，只需使用检测对象的属性即可。

import glob
import fiftyone as fo

images_patt = "/path/to/images/*"

# Ex: your custom label format
annotations = {
    "/path/to/images/000001.jpg": [
        {"bbox": ..., "label": ..., "mask": ...},
        ...
    ],
    ...
}

# Create samples for your data
samples = []
for filepath in glob.glob(images_patt):
    sample = fo.Sample(filepath=filepath)

    # Convert detections to FiftyOne format
    detections = []
    for objects in annotations[filepath]:
        label = obj["label"]

        # Bounding box coordinates should be relative values
        # in [0, 1] in the following format:
        # [top-left-x, top-left-y, width, height]
        bounding_box = obj["bbox"]

        # Boolean or 0/1 Numpy array 
        mask = obj["mask"]
        
        detection = fo.Detection(
            label=label,
            bounding_box=bounding_box,
            mask=mask,
        )

        detections.append(detection)

    # Store detections in a field name of your choice
    sample["ground_truth"] = fo.Detections(detections=detections)

    samples.append(sample)

# Create dataset
dataset = fo.Dataset("my-detection-dataset")
dataset.add_samples(samples)

预测可以与实际事实同时加载，也可以在将来的数据集和在稍后时间添加预测上进行迭代。

import fiftyone as fo

# Ex: your custom predictions format
predictions = {
    "/path/to/images/000001.jpg": [
        {"bbox": ..., "label": ..., "mask": ..., "score": ...},
        ...
    ],
    ...
}

# Add predictions to your samples
for sample in dataset:
    filepath = sample.filepath

    # Convert predictions to FiftyOne format
    detections = []
    for obj in predictions[filepath]:
        label = obj["label"]
        confidence = obj["score"]

        # Bounding box coordinates should be relative values
        # in [0, 1] in the following format:
        # [top-left-x, top-left-y, width, height]
        bounding_box = obj["bbox"]
        
        # Boolean or 0/1 Numpy array 
        mask = obj["mask"]

        detection = fo.Detection(
            label=label,
            bounding_box=bounding_box,
            confidence=confidence,
        )

        detection["mask"] = mask

        detections.append(detection)

    # Store detections in a field name of your choice
    sample["predictions"] = fo.Detections(detections=detections)

    sample.save()

注意，解析自定义标签格式的确切结构将取决于存储数据的方式。这只是一个例子，它将标签存储在按媒体文件键键的字典中。您可能首先需要解析和转换标签，例如，将其转换为预期的边框格式。