pywren和twint - Tweet下载

Question

下面的代码获取用户名，并从给定的日期刮取他们的twitter历史记录

import pandas as pd
import twint
import pywren

def scrape_user(username):
    c = twint.Config()
    c.Username = username
    c.Lang = 'es'
    c.Since = '2021-04-28'
    c.Hide_output = True
    c.Pandas = True

    twint.run.Search(c)

    return twint.storage.panda.Tweets_df

当我运行这个函数时，我得到了预期的结果，即Pandas数据，例如scrape_user("DeLaCalleHum")。但是，当我使用pywren时(甚至在一个用户名上)

pwex = pywren.default_executor()
futures = pwex.map(scrape_user, "DeLaCalleHum")
tweet_list = pywren.get_all_results(futures)

我知道这个错误。

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
<ipython-input-31-15f9e00ead75> in <module>
----> 1 wc_list = pywren.get_all_results(futures)

~/macs30123/lib/python3.7/site-packages/pywren/wren.py in get_all_results(fs)
    117     """
    118     wait(fs, return_when=ALL_COMPLETED)
--> 119     return [f.result() for f in fs]

~/macs30123/lib/python3.7/site-packages/pywren/wren.py in <listcomp>(.0)
    117     """
    118     wait(fs, return_when=ALL_COMPLETED)
--> 119     return [f.result() for f in fs]

~/macs30123/lib/python3.7/site-packages/pywren/future.py in result(self, timeout, check_only, throw_except, storage_handler)
    146         if self._state == JobState.error:
    147             if throw_except:
--> 148                 raise self._exception
    149             else:
    150                 return None

OSError: [Errno 28] No space left on device

我做错了什么？我很感谢你的帮助。

Answer 1

过了一段时间我找到了答案。我可以在PyWren中自动并行这些函数调用，只要我在IAM中将ComprehendFullAccess策略添加到pywren_exec_role_1角色中。