将url列的值逐个传递给Python中的web爬虫代码

Question

基于this link的应答代码，我能够创建一个新的列：df['url'] = 'https://www.cspea.com.cn/list/c01/' + df['projectCode']。

下一步，我希望将url列的值传递给下面的代码，并将所有已抓取的内容附加为dataframe。

import urllib3
import requests
from bs4 import BeautifulSoup
import pandas as pd

url = "https://www.cspea.com.cn/list/c01/gr2021bj1000186" # url column's values should be passed here one by one
soup = BeautifulSoup(requests.get(url, verify=False).content, "html.parser")
index, data = [], []
for th in soup.select(".project-detail-left th"):
    h = th.get_text(strip=True)
    t = th.find_next("td").get_text(strip=True)
    index.append(h)
    data.append(t)

df = pd.DataFrame(data, index=index, columns=["value"])
print(df) 

我怎么能用Python做到这一点呢？谢谢。

更新：

import requests
from bs4 import BeautifulSoup
import pandas as pd

df = pd.read_excel('items_scraped.xlsx')

data = []
urls =  df.url.tolist()
for url_link in urls:

    url = url_link
    # url = "https://www.cspea.com.cn/list/c01/gr2021bj1000186"
    soup = BeautifulSoup(requests.get(url, verify=False).content, "html.parser")
    
    index, data = [], []
    for th in soup.select(".project-detail-left th"):
        h = th.get_text(strip=True)
        t = th.find_next("td").get_text(strip=True)
        index.append(h)
        data.append(t)
    
    df = pd.DataFrame(data, index=index, columns=["value"])
    df = df.T
    df.reset_index(drop=True, inplace=True)
    print(df)

df.to_excel('result.xlsx', index = False)

但是它只将一行保存到excel文件中。

Answer 1

您需要将循环中生成的dfs组合起来。您可以将它们添加到列表中，然后调用列表上的pd.concat。

import requests
from bs4 import BeautifulSoup
import pandas as pd

df = pd.read_excel('items_scraped.xlsx')

# data = []
urls =  df.url.tolist()
dfs = []

for url_link in urls:

    url = url_link
    # url = "https://www.cspea.com.cn/list/c01/gr2021bj1000186"
    soup = BeautifulSoup(requests.get(url, verify=False).content, "html.parser")
    
    index, data = [], []
    for th in soup.select(".project-detail-left th"):
        h = th.get_text(strip=True)
        t = th.find_next("td").get_text(strip=True)
        index.append(h)
        data.append(t)
    
    df = pd.DataFrame(data, index=index, columns=["value"])
    df = df.T
    df.reset_index(drop=True, inplace=True)
    print(df)
    dfs.append(df)

df = pd.concat(dfs)
df.to_excel('result.xlsx', index = False)

Answer 2

使用

 urls =  df.url.tolist()

创建一个url列表，然后使用f字符串迭代它们，将每个url插入到基本url中。