抓取句柄302重定向

Question

我正在尝试用一个粗糙的CrawlSpider抓取一个网站，问题是这个网站一直在随机地重定向我，这意味着一个url有时可能会加载，有时会被重定向到某个页面。我试着改变我的用户代理，尝试通过创建一个类似于浏览器发送的http头来模仿浏览器的行为，即使我使用crawlera发送请求，也没有解决我的问题。如果有人能帮我度过难关，我将不胜感激

控制台：

2017-11-06 02:11:14 [scrapy.core.engine] INFO: Spider opened
2017-11-06 02:11:14 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-11-06 02:11:14 [scrapy.extensions.telnet] DEBUG: Telnet console listening on 127.0.0.1:6023
2017-11-06 02:11:15 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET http://www.sears.com/en_intnl/dap/shopping-tourism.html> from <GET http://www.sears.com/en_intnl/dap/shopping-tourism.html>
2017-11-06 02:11:22 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.sears.com/en_intnl/dap/shopping-tourism.html> (referer: None)
2017-11-06 02:11:27 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET http://www.sears.com/en_us/botmanagement.html> from <GET http://www.sears.com/en_us/sitemap.html>
2017-11-06 02:11:34 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.sears.com/en_us/botmanagement.html> (referer: http://www.sears.com/en_intnl/dap/shopping-tourism.html)
2017-11-06 02:11:34 [scrapy.dupefilters] DEBUG: Filtered duplicate request: <GET http://www.sears.com/gifts/b-1020009> - no more duplicates will be shown (see DUPEFILTER_DEBUG to show all duplicates)
2017-11-06 02:11:39 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.sears.com/seasonal-christmas/b-1100100> (referer: http://www.sears.com/en_intnl/dap/shopping-tourism.html)
2017-11-06 02:11:46 [scrapy.core.engine] DEBUG: Crawled (200) <GET http://www.sears.com/toys-games/b-1020010> (referer: http://www.sears.com/en_intnl/dap/shopping-tourism.html)
2017-11-06 02:11:52 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET http://www.sears.com/en_us/botmanagement.html> from <GET http://www.sears.com/home-decor-decorative-accents/b-1348893716>
2017-11-06 02:11:58 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET http://www.sears.com/en_us/botmanagement.html> from <GET http://www.sears.com/tvs-electronics-home-theater-audio-musical-instruments-guitars-string-instruments/b-5000861>
2017-11-06 02:12:03 [scrapy.downloadermiddlewares.redirect] DEBUG: Redirecting (302) to <GET http://www.sears.com/en_us/botmanagement.html> from <GET http://www.sears.com/tvs-electronics-gaming/b-1347529268>

Answer 1

如果你没有太多的代理：

• 当您使用parse解析html
  • 时，请使用if response.url == """http://www.sears.com/en_us/botmanagement.html""":来检测您是否被重定向到了reCAPCHA页面。
  • 使用Selenium和Scrapy(Selenium可以直接控制浏览器，因此您将能够观察整个抓取过程，并手动降低抓取速度以防止爬网爬虫Scrapy

​

• Gather proxies

• 公共代理

​