并行请求处理python + lambda

Question

我有一个大约80万个ID的列表，对于每个ID，我应该向api发送一个请求，获取数据并插入到我的数据库中。

当前流程是循环每个项目并等待数小时:/

使用python + lambda执行此任务的聪明方法是什么？

Answer 1

您可以在python中使用允许并行化进程的concurrent.futures：

https://docs.python.org/3/library/concurrent.futures.html

定义您的worker函数：

import time

def worker(id):
    print(f"start worker {id}")
    time.sleep(1)
    print(f"end worker {id}")

并行化：

currs = ThreadPoolExecutor(max_workers=5) # If you don't want more than 5 parallelized execution of your function

for i in range(20): # Assuming you want to submit 20 requests
    curr_future_result = currs.submit(worker, i)

如果你想执行api调用，你可以这样做：

import requests

urls = [.......]

def worker(url):
    requests.get(url)
    
currs = ThreadPoolExecutor(max_workers=5)

for url in urls:
    curr_future_result = currs.submit(worker, url)