scrapy爬虫没有跟随下一个链接

Question

我正在使用scrapy从意大利国家警察那里收集新闻。我的问题是，即使我有一个规则集来找到“下一步”或意大利语中的"Successiva“按钮并遵循该链接，但刮取器没有遵循”下一步“链接。

这是我的代码。

from scrapy.spiders import Rule, CrawlSpider
from scrapy.linkextractors import LinkExtractor
from scrapy.selector import Selector
from items import ScrapyCrimeScraperItem
from string import replace

class ItalyScraper(CrawlSpider):
name = 'italy_crawler_test'
allowed_domains = ['poliziadistato.it']

start_urls = [
    'http://www.poliziadistato.it/archivio/category/1298/2015/',
    'http://www.poliziadistato.it/archivio/category/1298/2015/9/'

]

rules = (Rule(LinkExtractor(allow=('http://.*/articolo/view/*.....')), callback='parse_article', follow=False),
Rule(LinkExtractor(restrict_xpaths=("/html/body/div[@class='container'[1]/div[@class='row']/div[@class='col-md-6 col-md-push-3 padding0']/div[@class='trecolonne']/div[@class='center']/div[@class='bar']/ul[@class='paginazione']/li/a[contains(""@title,""'Successiva')]",)), follow=True))

# def generate_article_links(self, response):
#     for href in response.css('a'):
#         url = href.extract()
#         yield scrapy.Request(url, callback=self.parse_article)


def parse_article(self, response):
    self.logger.info('Hi, this is an item page! %s', response.url)
    item = ScrapyCrimeScraperItem()
    item['city'] = response.selector.css('h1').extract()[0]
    item['country'] = 'italy'
    item['site_link'] = response.url
    item['article_link'] = response.url
    item['article_raw_text'] = self.remove_carriage_returns(response.selector.css('.resetfont '
                                                     'p').extract(

    )[0])
    item['article_raw_date'] = response.selector.css('.data').extract()[0]
    item['article_translated_text'] = ''
    item['article_translated_date'] = ''
    item['article_raw_markup'] = ''
    item['crimes'] = ''
    item['locations'] = ''
    item['dateformat'] = ''
    item['reserved1'] = ''
    item['reserved2'] = ''

    yield item


def remove_carriage_returns(self,item):

    return(item.replace("\n", " "))

我查看了一些类似问题的其他回答，但我在第二条规则上使用了显式follow=True。我是否需要一个回调来生成一个新的请求--或者以下变量不应该负责生成这个新的请求？

Answer 1

我认为您刚刚弄乱了XPath表达式中的引号。改用更简单的方法：

//ul[@class="paginazione"]/li/a[contains(@title, "Successiva")]