将S3对象转换为层次(树)映射

Question

在这个任务中，我需要使用S3从AWS ExAws中提取对象，并以类似文件树的结构将其返回到前端。

从ExAws.S3.list_objects("bucket-name", prefix: "folder-path") |> ExAws.request!返回的数据

%{
  body: %{
    contents: [
      %{key: "file1.svg"},
      %{key: "FolderA/file1.txt"},
      %{key: "FolderA/file2.txt"},
      %{key: "FolderB/Layer1/file1.csv"},
      %{key: "FolderB/Layer1/file2.csv"},
      %{key: "FolderB/Layer1/GroupA/file1.pdf"},
      %{key: "FolderB/Layer1/GroupB/file1.jpg"},
      %{key: "FolderB/Layer2/file1.png"}
    ]
  }
}

我要把它转换成这张地图。

%{
  "FolderA" => %{
    "file1.txt" => "FolderA/file1.txt",
    "file2.txt" => "FolderA/file2.txt"
  },
  "FolderB" => %{
    "Layer1" => %{
      "GroupA" => %{
        "file1.pdf" => "FolderB/Layer1/GroupA/file1.pdf"
      },
      "GroupB" => %{
        "file1.jpg" => "FolderB/Layer1/GroupB/file1.jpg"
      },
      "file1.csv" => "FolderB/Layer1/file1.csv",
      "file2.csv" => "FolderB/Layer1/file2.csv"
    },
    "Layer2" => %{"file1.png" => "FolderB/Layer2/file1.png"}
  },
  "file1.svg" => "file1.svg"
}

我找到了这个npm包，S3-树，但是它为每个‘文件夹’做了一个请求。我需要的只是纯粹的算法。

这些与蟒蛇有关：Python-如何将S3文件夹转换为JSON层次结构？，S3-树。把这些转化成长生不老药会有帮助。

Answer 1

我在这里再次为长生药最被低估的特性Access行为做广告。

Access.key/2是你在这里的朋友。

data
|> get_in([:body, :contents, Access.all(), :key])
|> Enum.reduce(%{}, fn e, acc ->
  put_in(acc, Enum.map(String.split(e, "/"), &Access.key(&1, %{})), e)
end)

就这样。

Answer 2

好的递归挑战

我对ex_aws_s3并不熟悉，所以可能已经有了一些功能，但下面是我的尝试：

defmodule S3Tree do
  def parse(data) do
    Enum.reduce(data.body.contents, %{}, fn %{key: full_path}, acc ->
      tree = traverse(String.split(full_path, "/"), full_path)
      merge(nil, acc, tree)
    end)
  end

  defp traverse([file], full_path), do: %{file => full_path}
  defp traverse([dir | rest], full_path), do: %{dir => traverse(rest, full_path)}

  defp merge(_key, map1, map2), do: Map.merge(map1, map2, &merge/3)
end

用法：

data = %{
  body: %{
    contents: [
      %{key: "file1.svg"},
      %{key: "FolderA/file1.txt"},
      %{key: "FolderA/file2.txt"},
      %{key: "FolderB/Layer1/file1.csv"},
      %{key: "FolderB/Layer1/file2.csv"},
      %{key: "FolderB/Layer1/GroupA/file1.pdf"},
      %{key: "FolderB/Layer1/GroupB/file1.jpg"},
      %{key: "FolderB/Layer2/file1.png"}
    ]
  }
}

S3Tree.parse(data)

输出：

%{
  "FolderA" => %{
    "file1.txt" => "FolderA/file1.txt",
    "file2.txt" => "FolderA/file2.txt"
  },
  "FolderB" => %{
    "Layer1" => %{
      "GroupA" => %{"file1.pdf" => "FolderB/Layer1/GroupA/file1.pdf"},
      "GroupB" => %{"file1.jpg" => "FolderB/Layer1/GroupB/file1.jpg"},
      "file1.csv" => "FolderB/Layer1/file1.csv",
      "file2.csv" => "FolderB/Layer1/file2.csv"
    },
    "Layer2" => %{"file1.png" => "FolderB/Layer2/file1.png"}
  },
  "file1.svg" => "file1.svg"
}