awswrangler:尝试读取表时无法启动新线程

Question

我正在尝试访问AWS存储桶中的表。当我尝试使用awswrangler.read_parquet函数访问它时，我得到一个错误，说我不能访问这个文件，因为我不能创建新的线程。在等待30min+之后，我通常能够访问该文件，但这并不能告诉我如何解决问题。以下是有关该命令的更多详细信息：

   aws_df = wr.s3.read_parquet(path=self._filepath, **self._load_args)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/awswrangler/s3/_read_parquet.py", line 721, in read_parquet
    read_func=_read_parquet, paths=paths, version_ids=versions, use_threads=use_threads, kwargs=args
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/awswrangler/s3/_read.py", line 145, in _read_dfs_from_multiple_paths
    return list(df for df in executor.map(partial_read_func, paths, versions))
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/awswrangler/s3/_read.py", line 145, in <genexpr>
    return list(df for df in executor.map(partial_read_func, paths, versions))
  File "/home/ec2-user/anaconda3/lib/python3.7/concurrent/futures/_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "/home/ec2-user/anaconda3/lib/python3.7/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/home/ec2-user/anaconda3/lib/python3.7/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/home/ec2-user/anaconda3/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/awswrangler/s3/_read_parquet.py", line 495, in _read_parquet
    version_id=version_id,
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/awswrangler/s3/_read_parquet.py", line 440, in _read_parquet_file
    source=f, read_dictionary=categories
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/awswrangler/s3/_read_parquet.py", line 40, in _pyarrow_parquet_file_wrapper
    return pyarrow.parquet.ParquetFile(source=source, read_dictionary=read_dictionary)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/pyarrow/parquet.py", line 201, in __init__
    read_dictionary=read_dictionary, metadata=metadata)
  File "pyarrow/_parquet.pyx", line 1021, in pyarrow._parquet.ParquetReader.open
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/awswrangler/s3/_fs.py", line 569, in read
    self._fetch(self._loc, self._loc + length)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/awswrangler/s3/_fs.py", line 376, in _fetch
    self._cache = self._fetch_range_proxy(self._start, self._end)
  File "/home/ec2-user/anaconda3/lib/python3.7/site-packages/awswrangler/s3/_fs.py", line 359, in _fetch_range_proxy
    itertools.repeat(self._version_id),
  File "/home/ec2-user/anaconda3/lib/python3.7/concurrent/futures/_base.py", line 575, in map
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
  File "/home/ec2-user/anaconda3/lib/python3.7/concurrent/futures/_base.py", line 575, in <listcomp>
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
  File "/home/ec2-user/anaconda3/lib/python3.7/concurrent/futures/thread.py", line 160, in submit
    self._adjust_thread_count()
  File "/home/ec2-user/anaconda3/lib/python3.7/concurrent/futures/thread.py", line 181, in _adjust_thread_count
    t.start()
  File "/home/ec2-user/anaconda3/lib/python3.7/threading.py", line 847, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

Answer 1

我能够通过在我的命令之间添加睡眠来解决这个问题。

import time
time.sleep(10)