抓取抓取中的链接

Question

我试图建立一个广泛的连续爬虫，我能够提取链接，但我无法抓取他们和提取这些链接。该项目的最终目标是抓取.au域并将其根URL添加到数据库中。

class Crawler (scrapy.Spider):
    name = "crawler"
    rules = (Rule(LinkExtractor(allow='.com'), callback='parse_item')) 
    #This will be changed to allow .au before deployment to only crawl .au sites.

    start_urls = [
        "http://quotes.toscrape.com/",
    ]

    def parse(self, response):
        urls = response.xpath("//a/@href")
        for u in urls:
            l = ItemLoader(item=Link(), response=response)
            l.add_xpath('url', './/a/@href')
            return l.load_item()

另一个问题是，对于内部链接，它是添加一个相对url路径，而不是绝对路径。我试过用这一节来修复它。

urls = response.xpath("//a/@href")
        for u in urls:

items.py文件：

class Link(scrapy.Item):
    url = scrapy.Field()
    pass

Answer 1

我设法弄清楚了，我在下面贴出了基本代码，以帮助将来有同样问题的人。

from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

#Create a list of sites not to crawl. 
#Best to read this from a file containing top 100 sites for example.
denylist = [
    'google.com',
    'yahoo.com',
    'youtube.com'
]

class Crawler (CrawlSpider): #For broad crawl you need to use "CrawlSpider"
    name = "crawler"
    rules = (Rule(LinkExtractor(allow=('.com', ), 
    deny=(denylist)), follow=True, callback='parse_item'),)

    start_urls = [
        "http://quotes.toscrape.com",
    ]


    def parse_item(self, response):
        # self.logger.info('LOGGER %s', response.url)  
        # use above to log and see info in the terminal

        yield {
            'link': response.url
        }

Answer 2

ItemLoaders对于创建需要Processors的项对象非常有用。从您的代码中，我不认为有必要使用这些。您可以简单地yield一个Request对象。

您可以去掉Link类(附带注意:当块中没有其他东西时，pass被用作占位符)。所以代码中的pass是没有意义的)

 def parse(self, response):
        urls = response.xpath("//a/@href")
        for u in urls:
           yield scrapy.Request(u, callback=self.your_callback_method)

希望它能帮上忙