在juputerlab上运行时，dask将文件存储在哪里

Question

我在jupyterlab上运行dask。我正在尝试在存储python文件的主目录中保存一些文件，它正在正常运行，但我找不到我的文件被保存的位置。所以我在主目录中创建了一个名为output的文件夹来保存文件，但是当我在其中保存文件时，我得到了以下错误：

PermissionError: [Errno 13] Permission denied: b'/home/jovyan/Output/20190101_0200'

下面是我的尝试：

from dask_gateway import Gateway
gateway = Gateway(
    address="http://traefik-pangeo-dask-gateway/services/dask-gateway",
    public_address="https://pangeo.aer-gitlab.com/services/dask-gateway",
    auth="jupyterhub",
)
options = gateway.cluster_options()
options

cluster = gateway.new_cluster(
    cluster_options=options,
)
cluster.adapt(minimum=10, maximum=50)
client = cluster.get_client()
cluster
client

def get_turb(file, name):
    
    d=[name[0:4],name[4:6],name[6:8],name[9:11],name[11:13]] 
    f_zip = gzip.decompress(file)

    yr=d[0]
    mo=d[1]
    da=d[2]
    hr=d[3]
    mn=d[4]
    
    fs = s3fs.S3FileSystem(anon=True)

    period = pd.Period(str(yr)+str('-')+str(mo)+str('-')+str(da), freq='D')
    # period.dayofyear
    dy=period.dayofyear

    cc=[7,8,9,10,11,12,13,14,15,16]  #look at the IR channels only for now
    dat = xr.open_dataset(f_zip)
    dd=dat[['recNum','trackLat','trackLon','altitude','maxEDR']]
    dd=dd.to_dataframe()
    dd = dd.sort_values(by=['maxEDR'])
    dd = dd.dropna()
    dd['num'] = np.arange(len(dd))
    dd.to_csv('Output/edr.csv') <----- Saving the file

edr_files = []

for i in range(2):
    print(names[i])
    s3_ds = dask.delayed(get_turb)(filedata[i], names[i])
    edr_files.append(s3_ds)

edr_files = dask.compute(*edr_files)

请告诉我我做错了什么，或者可能的解决方案是什么。

另外，当我尝试使用以下代码直接将文件保存在S3存储桶中时：

    s3.Bucket('temp').upload_file(file_name+'.zip', file_name+'.zip')

它抛出了这个错误：

distributed.protocol.pickle - INFO - Failed to serialize <function get_temp at 0x7f20a9cb8550>. Exception: cannot pickle '_thread.lock' object
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/worker.py in dumps_function(func)
   3319         with _cache_lock:
-> 3320             result = cache_dumps[func]
   3321     except KeyError:

/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/utils.py in __getitem__(self, key)
   1572     def __getitem__(self, key):
-> 1573         value = super().__getitem__(key)
   1574         self.data.move_to_end(key)

/srv/conda/envs/notebook/lib/python3.8/collections/__init__.py in __getitem__(self, key)
   1009             return self.__class__.__missing__(self, key)
-> 1010         raise KeyError(key)
   1011     def __setitem__(self, key, item): self.data[key] = item

KeyError: <function get_temp at 0x7f20a9cb8550>

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
/srv/conda/envs/notebook/lib/python3.8/site-packages/distributed/protocol/pickle.py in dumps(x, buffer_callback, protocol)
     52                 buffers.clear()
---> 53                 result = cloudpickle.dumps(x, **dump_kwargs)
     54         elif not _always_use_pickle_for(x) and b"__main__" in result:

/srv/conda/envs/notebook/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py in dumps(obj, protocol, buffer_callback)
     72             )
---> 73             cp.dump(obj)
     74             return file.getvalue()

/srv/conda/envs/notebook/lib/python3.8/site-packages/cloudpickle/cloudpickle_fast.py in dump(self, obj)
    562         try:
--> 563             return Pickler.dump(self, obj)
    564         except RuntimeError as e:

TypeError: cannot pickle '_thread.lock' object

Answer 1

尝试使用s3fs而不是boto3在S3上载文件。这可能行得通。

Answer 2

看来你在码头上运行了dask和jupyterlab？

也许你应该添加一些像fellowing这样的标志：

--user root \
-v new-longhorn-volume:/home/jovyan \
-e CHOWN_HOME=yes \
-e CHOWN_HOME_OPTS='-R' \