Web从Tripadvisor抓取链接列表

Question

我正在尝试创建一个webscraper，它将从网站示例网站返回指向单个对象的链接列表。我编写的代码获取页面列表，并将链接列表返回到每个景点，但以错误的方式(链接不是一个接一个)：

​

谁能帮我更正一下这段代码，这样它就可以把下面的链接列出来了吗？

​

我将非常感谢您的帮助。

我的代码：

import requests
from bs4 import BeautifulSoup

header = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36',
}

restaurantLinks = open('pages.csv')
print(restaurantLinks)
urls = [url.strip() for url in restaurantLinks.readlines()]


restlist = []
for link in urls:
    print("Opening link:"+str(link))
    response=requests.get(link, headers=header)
    soup = BeautifulSoup(response.text, 'html.parser')

    productlist = soup.find_all('div', class_='cNjlV')
    print(productlist)

    productlinks =[]
    for item in productlist:
        for link in item.find_all('a', href=True):
            productlinks.append('https://www.tripadvisor.com'+link['href'])

    print(productlinks)

    restlist.append(productlinks)

print(restlist)

df = pd.DataFrame(restlist)
df.to_csv('links.csv')

Answer 1

不要将append()元素添加到列表中，而是尝试将其放在extend()中：

restlist.extend(productlinks)

示例

import requests
from bs4 import BeautifulSoup

header = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36',
}
urls = ['https://www.tripadvisor.com/Attractions-g187427-Activities-oa60-Spain.html']
restlist = []

for link in urls:
    print("Opening link:"+str(link))
    response=requests.get(link, headers=header)
    soup = BeautifulSoup(response.text, 'html.parser')
    
    restlist.extend(['https://www.tripadvisor.com'+a['href'] for a in soup.select('a:has(h3)')])
    
df = pd.DataFrame(restlist)
df.to_csv('links.csv', index=False)