如何在BeautifulSoup python中获取实时进度条？

Question

我有下面的代码，代码从像红泡这样的网站上抓取了一些数据。有时我收集了很多数据，我想知道代码中的实时进度……我尝试了进度条模块，但我没有得到我想要的……

import requests
from bs4 import BeautifulSoup

re = requests.get('https://www.redbubble.com/i/iphone-case/What-A-Time-To-Be-Alive-by-DinoMike/36490886.RIOBD')
        
src = re.content

soup = BeautifulSoup(src, "html.parser")

tags = soup.find_all("span", {"class" : "styles__children--21o3C"})

print(tags)

Answer 1

如果您有多个页面可供请求，这里有一个很酷的库tqdm，它显示了一个进度条。

import requests
from bs4 import BeautifulSoup
from tqdm import tqdm

# set of target URLs
urls = [
    "https://www.redbubble.com/i/iphone-case/What-A-Time-To-Be-Alive-by-DinoMike/36490886.RIOBD",
    ...
]
set_tags = []
# go through the list
for url in tqdm(urls):
    # get request
    soup = BeautifulSoup(requests.get(url).content, "html.parser")
    tags = soup.find_all("span", {"class": "styles__children--21o3C"})
    set_tags.append(tags)