当试图在爬行的urls上迭代时，刮伤蜘蛛不起作用。

Question

我是个有刮痕的新手。当我试图从论坛上抓取数据时，我的蜘蛛没有正常工作。当我运行我的蜘蛛，它只给我打印的网址和停止后。因此，我认为问题在于两个函数解析和parse_data的兼容性，但我可能错了。这是我的代码：

import scrapy, time


class ForumSpiderSpider(scrapy.Spider):
    name = 'forum_spider'
    allowed_domains = ['visforvoltage.org/latest_tech/']
    start_urls = ['http://visforvoltage.org/latest_tech//']

    def parse(self, response):
        for href in response.css(r"tbody a[href*='/forum/']::attr(href)").extract():
           url = response.urljoin(href)
           print(url)
           req = scrapy.Request(url, callback=self.parse_data)
           time.sleep(10)
           yield req

    def parse_data(self, response):
        for url in response.css('html').extract():
           data = {}
           data['name'] = response.css(r"div[class='author-pane-line author-name'] span[class='username']::text").extract()
           data['date'] = response.css(r"div[class='forum-posted-on']:contains('-') ::text").extract()
           data['title'] = response.css(r"div[class='section'] h1[class='title']::text").extract()
           data['body'] = response.css(r"div[class='field-items'] p::text").extract()
           yield data
       

        next_page = response.css(r"li[class='pager-next'] a[href*='page=']::attr(href)").extract()
        if next_page:
            yield scrapy.Request(
                response.urljoin(next_page),
                callback=self.parse)

这是输出：

2020-07-23 23:09:58 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'visforvoltage.org': <GET https://visforvoltage.org/forum/14521-aquired-a123-m1-cells-need-charger-and-bms>
https://visforvoltage.org/forum/14448-battery-charger-problems
https://visforvoltage.org/forum/14191-vectrix-trickle-charger
https://visforvoltage.org/forum/14460-what-epoxy-would-you-recommend-loose-magnet-repair
https://visforvoltage.org/forum/14429-importance-correct-grounding-and-well-built-plugs
https://visforvoltage.org/forum/14457-147v-charger-24v-lead-acid-charger-and-dying-vectrix-cells
https://visforvoltage.org/forum/6723-lithium-safety-e-bike
https://visforvoltage.org/forum/11488-how-does-24v-4-wire-reversible-motor-work
https://visforvoltage.org/forum/14444-new-sevcon-gen-4-80v-sale
https://visforvoltage.org/forum/14443-new-sevcon-gen-4-80v-sale
https://visforvoltage.org/forum/12495-3500w-hub-motor-question-about-real-power-and-breaker
https://visforvoltage.org/forum/14402-vectrix-vx-1-battery-pack-problem
https://visforvoltage.org/forum/14068-vectrix-trickle-charger
https://visforvoltage.org/forum/2931-drill-motors
https://visforvoltage.org/forum/14384-help-repairing-gio-hub-motor-freewheel-sprocket
https://visforvoltage.org/forum/14381-zev-charger
https://visforvoltage.org/forum/8726-performance-unite-my1020-1000w-motor
https://visforvoltage.org/forum/7012-controler-mod-veloteq
https://visforvoltage.org/forum/14331-scooter-chargers-general-nfpanec
https://visforvoltage.org/forum/14320-charging-nissan-leaf-cells-lifepo4-charger
https://visforvoltage.org/forum/3763-newber-needs-help-new-gift-kollmorgan-hub-motor
https://visforvoltage.org/forum/14096-european-bldc-controller-seller
https://visforvoltage.org/forum/14242-lithium-bms-vs-manual-battery-balancing
https://visforvoltage.org/forum/14236-mosfet-wiring-ignition-key
https://visforvoltage.org/forum/2007-ok-dumb-question-time%3A-about-golf-cart-controllers
https://visforvoltage.org/forum/10524-my-mf70-recommended-powerpoles-arrived-today
https://visforvoltage.org/forum/9460-how-determine-battery-capacity
https://visforvoltage.org/forum/7705-tricking-0-5-v-hall-effect-throttle
https://visforvoltage.org/forum/13446-overcharged-lead-acid-battery-what-do
https://visforvoltage.org/forum/14157-reliable-high-performance-battery-enoeco-bt-p380
https://visforvoltage.org/forum/2702-hands-test-48-volt-20-ah-lifepo4-pack-ping-battery
https://visforvoltage.org/forum/14034-simple-and-cheap-ev-can-bus-adaptor
https://visforvoltage.org/forum/13933-zivan-ng-3-charger-specs-and-use
https://visforvoltage.org/forum/13099-controllers
https://visforvoltage.org/forum/13866-electric-motor-werks-demos-25-kilowatt-diy-chademo-leaf
https://visforvoltage.org/forum/13796-motor-theory-ac-vs-bldc
https://visforvoltage.org/forum/6184-bypass-bms-lifepo4-good-idea-or-not
https://visforvoltage.org/forum/13763-positive-feedback-kelly-controller
https://visforvoltage.org/forum/13764-any-users-smart-battery-drop-replacement-zapino-and-others
https://visforvoltage.org/forum/13760-contactor-or-fuse-position-circuit-rules-why
https://visforvoltage.org/forum/13759-contactor-or-fuse-position-circuit-rules-why
https://visforvoltage.org/forum/12725-repairing-lithium-battery-pack
https://visforvoltage.org/forum/13752-questions-sepex-motor-theory
https://visforvoltage.org/forum/13738-programming-curtis-controller-software
https://visforvoltage.org/forum/13741-making-own-simple-controller
https://visforvoltage.org/forum/12420-idea-charging-electric-car-portably-wo-relying-electricity-infrastructure
2020-07-23 23:17:28 [scrapy.extensions.logstats] INFO: Crawled 2 pages (at 2 pages/min), scraped 0 items (at 0 items/min)
2020-07-23 23:17:28 [scrapy.core.engine] INFO: Closing spider (finished)

在我看来，它没有迭代这些链接并从它们收集数据。那是什么原因呢？我真的很感激你的帮助。谢谢!

Answer 1

这是我的工作。

import scrapy, time


class ForumSpiderSpider(scrapy.Spider):
    name = 'forum_spider'
    allowed_domains = ['visforvoltage.org/latest_tech/']
    start_urls = ['http://visforvoltage.org/latest_tech/']

    def parse(self, response):
        for href in response.css(r"tbody a[href*='/forum/']::attr(href)").extract():
            url = response.urljoin(href)
            req = scrapy.Request(url, callback=self.parse_data, dont_filter=True)
            yield req

    def parse_data(self, response):
        for url in response.css('html'):
            data = {}
            data['name'] = url.css(r"div[class='author-pane-line author-name'] span[class='username']::text").extract()
            data['date'] = url.css(r"div[class='forum-posted-on']:contains('-') ::text").extract()
            data['title'] = url.css(r"div[class='section'] h1[class='title']::text").extract()
            data['body'] = url.css(r"div[class='field-items'] p::text").extract()
            yield data

        next_page = response.css(r"li[class='pager-next'] a[href*='page=']::attr(href)").extract()
        if next_page:
            yield scrapy.Request(
                response.urljoin(next_page),
                callback=self.parse)

Answer 2

问题可能是请求正在被过滤，因为它们不是允许的域的一部分。

allowed_domains = ['visforvoltage.org/latest_tech/']

新请求网址：

https://visforvoltage.org/forum/14448-battery-charger-problems
https://visforvoltage.org/forum/14191-vectrix-trickle-charger
...

因为请求是给url visforvoltage.org/forum/的，而不是visforvoltage.org/latest_tech/的

可以完全删除允许的域属性，或更改为：

allowed_domains = ['visforvoltage.org']

这将使他们在页面上爬行，您将在日志中的这一行中看到一个不同的值：

2020-07-23 23:17:28 [scrapy.extensions.logstats] INFO: Crawled 2 pages (at 2 pages/min), scraped 0 items (at 0 items/min)

然而，解析中的选择器似乎不正确.

此选择器将选择整个页面，extract()方法将将其作为列表返回。因此，您将有一个list，其中只有一个字符串，它由页面的所有HTML组成。

response.css('html').extract()

您可以阅读关于选择器和getall()/extract()方法这里的更多信息。