文章/答案/技术大牛

发布

社区首页 >问答首页 >利用刮除和python抓取Tripadvisor景点

问利用刮除和python抓取Tripadvisor景点
EN

Stack Overflow用户

提问于 2022-06-06 19:36:32

回答 2查看 93关注 0票数 0

我试图刮TripAdvisor的吸引力，但我无法得到每个景点的名称和地址。我怀疑我写了product.css(.)(有麻烦事吗？)

有人能告诉我如何更正密码以得到每个景点的名称和地址吗？

我现在的代码是：

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'https://www.tripadvisor.com/Attractions-g187427-Activities-oa90-Spain'
    ]

    def parse(self, response):
        for link in response.css('.EsZYd a::attr(href)'):
            yield response.follow(link.get(), callback=self.parse_categories)

    def parse_categories(self, response):
        products = response.css('div.eeqnt')
        for product in products:
            yield {
                'name' : product.css('h1.WlYyy cPsXC GeSzT::text').get().strip(),
                'address' : product.css('span.WlYyy cacGK Wb::text').get().strip(),
            }

更新的代码(从列表中的每一页中的每一页导出信息)：

import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'https://www.tripadvisor.com/Attractions-g274862-Activities-a_allAttractions.true-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa30-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa60-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa90-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa120-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa150-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa180-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa210-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa240-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa270-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa300-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa330-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa360-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa390-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa420-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa450-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa480-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa510-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa540-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa570-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa600-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa630-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa660-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa690-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa720-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa750-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa780-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa810-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa840-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa870-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa900-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa930-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa960-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa990-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1020-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1050-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1080-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1110-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1140-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1170-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1200-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1230-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1260-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1290-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1320-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1350-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1380-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1410-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1440-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1470-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1500-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1530-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1560-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1590-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1620-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1650-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1680-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1710-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1740-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1770-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1800-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1830-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1860-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1890-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1920-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1950-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa1980-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2010-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2040-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2070-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2100-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2130-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2160-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2190-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2220-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2250-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2280-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2310-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2340-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2370-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2400-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2430-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2460-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2490-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2520-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2550-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2580-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2610-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2640-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2670-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2700-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2730-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2760-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2790-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2820-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2850-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2880-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2910-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2940-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa2970-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa3000-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa3030-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa3060-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa3090-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa3120-Slovenia.html',
        'https://www.tripadvisor.com/Attractions-g274862-Activities-oa3150-Slovenia.html'
    ]

    def parse(self, response):
            for link in response.css('.EsZYd a::attr(href)').getall():
                yield response.follow(link, callback=self.parse_categories)

    def parse_categories(self, response):
        yield {
            'name': response.css('h1.WlYyy.cPsXC.GeSzT::text').get(),
            'reviews': response.xpath('(//*[@class="cfIVb"])[1]//text()').getall(),
            'address': response.xpath('(//*[@class="dGWve"])//text()').getall(),
            'url': response.url,
        }

css

json

web-scraping

scrapy

python

回答 2

Stack Overflow用户

发布于 2022-06-06 22:28:41

它实际上与python无关，但是css-选择器。

CSS类应该用点而不是空格WlYyy.cPsXC.GeSzT分隔。

最好的建议是在开发工具条中使用铬。它将使您能够通过css选择器或xpath获得特定元素的路径，只需右键单击DOM树中的元素并选择“复制菜单项”。

避免使用类(尤其是没有语义意义的类)作为锚点。它们可能会在每一页之间或在时间上发生变化。

最好使用语义上有意义的节点，比如在您的例子中:标题的XPath看起来像这个//main//header//div[@data-automation="main_h1"]//h1。

票数 2

Stack Overflow用户

发布于 2022-06-07 00:22:01

不能在每个列表页面中使用for loop

from scrapy.crawler import CrawlerProcess
import scrapy

class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = [
        'https://www.tripadvisor.com/Attractions-g187427-Activities-oa90-Spain'
    ]

    def parse(self, response):
        for link in response.css('.EsZYd a::attr(href)').getall():
            #print(link)
            yield response.follow(link, callback=self.parse_categories)

    def parse_categories(self, response):
       
    
        yield {
            
            'name' : response.css('h1.WlYyy.cPsXC.GeSzT::text').get(),
            'address' :''.join(response.xpath('(//*[@class="hxQKk"])[1]//text()').getall()[:-1]),
            'url':response.url
            }
if __name__ == "__main__":
    process =CrawlerProcess(QuotesSpider)
    process.crawl()
    process.start()

票数 1

页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持

原文链接：

https://stackoverflow.com/questions/72522664

复制

相似问题

问利用刮除和python抓取Tripadvisor景点
EN

回答 2

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问利用刮除和python抓取Tripadvisor景点EN

回答 2

Stack Overflow用户

Stack Overflow用户

社区

活动

圈层

关于

腾讯云开发者

热门产品

热门推荐

更多推荐

问利用刮除和python抓取Tripadvisor景点
EN