抓痕爬虫

Question

Scrapy 2.0.1，scrapy_crawlera 1.7.0。我认为scrapy_crawlera应该以不同的方式访问元数据(https://github.com/scrapy/scrapy/issues/3516)

2020-04-02 06:02:36 [scrapy.core.engine] INFO: Spider opened
2020-04-02 06:02:36 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-04-02 06:02:36 [officedepot] INFO: Spider opened: officedepot
2020-04-02 06:02:36 [root] INFO: Using crawlera at http://proxy.crawlera.com:8010 (apikey: 0036675)
2020-04-02 06:02:36 [root] INFO: CrawleraMiddleware: disabling download delays on Scrapy side to optimize delays introduced by Crawlera. To avoid this behaviour you can use the CRAWLERA_PRESERVE_DELAY setting but keep in mind that this may slow down the crawl significantly
2020-04-02 06:02:36 [scrapy.extensions.httpcache] DEBUG: Using filesystem cache storage in /root/.scrapy/httpcache
2020-04-02 06:02:36 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-04-02 06:02:36 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.officedepot.com/a/products/9859127/Office-Depot-Brand-EverBind-D-Ring/>
Traceback (most recent call last):
  File "/root/venv/lib/python3.7/site-packages/scrapy/http/response/__init__.py", line 30, in meta
    return self.request.meta
AttributeError: 'NoneType' object has no attribute 'meta'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/venv/lib/python3.7/site-packages/twisted/internet/defer.py", line 1418, in _inlineCallbacks
    result = g.send(result)
  File "/root/venv/lib/python3.7/site-packages/scrapy/core/downloader/middleware.py", line 53, in process_response
    response = yield method(request=request, response=response, spider=spider)
  File "/root/venv/lib/python3.7/site-packages/scrapy_crawlera/middleware.py", line 192, in process_response
    retries = response.meta.get('crawlera_auth_retry_times', 0)
  File "/root/venv/lib/python3.7/site-packages/scrapy/http/response/__init__.py", line 33, in meta
    "Response.meta not available, this response "
AttributeError: Response.meta not available, this response is not tied to any request

Answer 1

只需将您的scrapy-crawlera版本更新到>= 1.7.2