在远程zarr存储上提供zarr.errors.GroupNotFoundError的open_mfdataset()

Question

我正在寻找使用xarray.open_mfdataset()读取远程zarr存储

我要买一台zarr.errors.GroupNotFoundError: group not found at path ''。在底部回溯。

import xarray as xr
import s3fs

fs = s3fs.S3FileSystem(anon=True)
uri = "s3://era5-pds/zarr/2020/12/data/eastward_wind_at_10_metres.zarr"
file = s3fs.S3Map(uri, s3=fs)
ds = xr.open_mfdataset(file, engine="zarr")

我可以使用xr.open_zarr打开它

ds = xr.open_zarr(file)

如果我将zarr商店下载到本地，它可以正常工作

import xarray as xr
import s3fs
fs = s3fs.S3FileSystem(anon=True)
fs.get("s3://era5-pds/zarr/2020/12/data/eastward_wind_at_10_metres.zarr/*", "eastward_wind_at_10_metres.zarr", recursive=True)
ds = xr.open_mfdataset("eastward_wind_at_10_metres.zarr", engine="zarr")

远程zarr存储上open_mfdataset的Trackback

>>> ds = xr.open_mfdataset(file, engine="zarr")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/api.py", line 948, in open_mfdataset
    datasets = [open_(p, **open_kwargs) for p in paths]
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/api.py", line 948, in <listcomp>
    datasets = [open_(p, **open_kwargs) for p in paths]
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/api.py", line 572, in open_dataset
    store = opener(filename_or_obj, **extra_kwargs, **backend_kwargs)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/xarray/backends/zarr.py", line 296, in open_group
    zarr_group = zarr.open_group(store, **open_kwargs)
  File "/Users/ray.bell/miniconda/envs/test_env/lib/python3.8/site-packages/zarr/hierarchy.py", line 1166, in open_group
    raise GroupNotFoundError(path)
zarr.errors.GroupNotFoundError: group not found at path ''

Answer 1

我认为open_mfdataset在你的path上做了一些glob，因为它希望处理多个路径(因为这是多个文件)。

下面的代码可以工作(前者可能只在master上运行)

ds = xr.open_mfdataset(uri, engine="zarr", backend_kwargs=dict(storage_options={'anon': True}))

ds = xr.open_mfdataset([file], engine="zarr")