错误:在multipe处理中超过最大递归深度

Question

我尝试使用ThreadPool运行multipe处理，并且我有错误maximum recursion depth exceeded。

这是我的密码：

def extract_all_social_link(bio_url):
    data = extract_all_social_link(bio_url)
    return data
def run_extract_all_social_link(df, max_count, displays_steps = 1000):
    tt = time.time()
    user_data = []
    with concurrent.futures.ThreadPoolExecutor(max_workers=NUM_WORKERS) as executor:
        future_to_url = dict()
        cnt = 0
        for _, row in df.iterrows():
            bio_url = row["bio_url"]
            if cnt > max_count:
                break
            if pd.isna(bio_url):
                continue
            future = executor.submit(extract_all_social_link, bio_url)
            future_to_url[future] = (bio_url)
            cnt += 1
        future_iter = concurrent.futures.as_completed(future_to_url)
        total = len(future_to_url)
        for cnt, future in tqdm(enumerate(future_iter), total=total):
            if (cnt+1) % displays_steps == 0:
                tt1 = time.time()
                print(f"{cnt+1} requests in {tt1 - tt:.3f} seconds")
                tt = tt1
            bio_url = future_to_url[future]
            try:
                data = future.result()
            except Exception as exc:
                print(f"{bio_url} generated an exception: {exc}")
        return user_data

`

这是错误：

generated an exception: maximum recursion depth exceeded

我怎么才能修好它？

Answer 1

甚至不要尝试更改递归限制配置。您必须用相同的arg extract_all_social_link()修改递归调用的bio_url代码。

def extract_all_social_link(bio_url):
    data = extract_all_social_link(bio_url)  # infinite recursion
    return data  # never reached !!

Answer 2

您可以使用https://docs.python.org/3/library/sys.html#sys.getrecursionlimit查看和增加这个限制，但这通常是不好的做法，因为它是防止堆栈溢出的一种安全措施。通过重构代码，找到一种避免如此多递归的方法。