使用父目录的抓取LinkExtractor抓取链接

Question

使用Scrapy中的基本CrawlerSpider，我正在尝试爬取页面。我要抓取的页面中的相关链接都以父目录符号..开头，而不是以完整的域开头。

例如，如果我从页面https://www.mytarget.com/posts/4/friendly-url开始，并且我想在/posts中抓取每个帖子，那么该页面上的相关链接将是：

'../55/post-name'
'../563/another-name'

而不是：

'posts/55/post-name'
'posts/563/another-name'

或者哪种方式更好：

'https://www.mytarget.com/posts/55/post-name'
'https://www.mytarget.com/posts/563/another-name'

从allowed_domains中删除mytarget.com似乎没有什么帮助。crawler在网站上找不到与..父目录链接引用匹配的新链接。

下面是我的代码：

import scrapy
from scrapy.linkextractors import LinkExtractor
from scrapy.spiders import CrawlSpider, Rule
from exercise_data_collector.items import Post

class MyCrawlerSpider(CrawlSpider):
    name = 'my_crawler'
    allowed_domains = ['mytarget.com']
    start_urls = ['https://www.mytarget.com/posts/4/friendly-url']

    rules = (
        Rule(LinkExtractor(allow=r'posts/[0-9]+/[0-9A-Za-z-_]+'), callback='parse_item', follow=True),
        Rule(LinkExtractor(allow=r'/posts\/[0-9]+\/[0-9A-Za-z-_]+'), callback='parse_item', follow=True),
        Rule(LinkExtractor(allow=r'/..\/[0-9]+\/[0-9A-Za-z-_]+'), callback='parse_item', follow=True),
    )

    def parse(self, response):
        links = self.le1.extract_links(response)

        item = Post()
        item["page_title"] = response.xpath('//title/text()').get()
        item["name"] = response.xpath("//div[@class='container']/div[@class='row']/div[1]/div[1]/text()[2]").get().replace('->','').strip()
        item['difficulty'] = response.xpath("//p[strong[contains(text(), 'Difficulty')]]/text()").get().strip()

        return item

我不确定如何配置正则表达式来获取相关链接，甚至测试正则表达式是否在regexr.com之外工作。

我怎么能像这样抓取页面呢？

Answer 1

我用这个正则表达式r'posts/[0-9]+/[A-Za-z-_]+'解决了这个问题

class MyCrawlerSpider(CrawlSpider):
    name = 'my_crawler'
    allowed_domains = ['mytarget.com']
    start_urls = ['https://www.mytarget.com/posts/4/friendly-url']

    rules = (
        Rule(LinkExtractor(allow=r'exercises/[0-9]+/[A-Za-z-_]+'), callback='parse_item', follow=True)
    )
    def parse(self, response):
        links = self.le1.extract_links(response)

        item = Post()
        item["page_title"] = response.xpath('//title/text()').get()
        item["name"] = response.xpath("//div[@class='container']/div[@class='row']/div[1]/div[1]/text()[2]").get().replace('->','').strip()
        item['difficulty'] = response.xpath("//p[strong[contains(text(), 'Difficulty')]]/text()").get().strip()

        return item

我确实遇到了一个递归问题，posts/12/page.html更改为posts/12/12/page.html ... posts/12/12/12/12/12/12/page.html。我认为这可能是他们网站上的一个错误。