如何通过Polipo over TOR连接到带有Scrapy的https站点？

Question

不完全确定这里的问题是什么。

运行Python 2.7.3和Scrapy 0.16.5

我已经创建了一个非常简单的Scrapy爬虫来测试连接到我的本地Polipo代理，这样我就可以通过TOR发送请求。我的爬虫的基本代码如下：

from scrapy.spider import BaseSpider

class TorSpider(BaseSpider):
    name = "tor"
    allowed_domains = ["check.torproject.org"]
    start_urls = [
        "https://check.torproject.org"
    ]

    def parse(self, response):
        print response.body

对于我的代理中间件，我定义了：

class ProxyMiddleware(object):
    def process_request(self, request, spider):
        request.meta['proxy'] = settings.get('HTTP_PROXY')

在我的设置文件中，我的HTTP_PROXY定义为HTTP_PROXY = 'http://localhost:8123'。

现在，如果我将开始URL更改为http://check.torproject.org，一切工作正常，没有问题。

如果我尝试在https://check.torproject.org上运行，我每次都会得到一个400 Bad Request错误(我也尝试过不同的https://站点，它们都有相同的问题)：

2013-07-23 21:36:18+0100 [scrapy] INFO: Scrapy 0.16.5 started (bot: arachnid)
2013-07-23 21:36:18+0100 [scrapy] DEBUG: Enabled extensions: LogStats, TelnetConsole, CloseSpider, WebService, CoreStats, SpiderState
2013-07-23 21:36:18+0100 [scrapy] DEBUG: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, RandomUserAgentMiddleware, ProxyMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, ChunkedTransferMiddleware, DownloaderStats
2013-07-23 21:36:18+0100 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2013-07-23 21:36:18+0100 [scrapy] DEBUG: Enabled item pipelines: 
2013-07-23 21:36:18+0100 [tor] INFO: Spider opened
2013-07-23 21:36:18+0100 [tor] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2013-07-23 21:36:18+0100 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2013-07-23 21:36:18+0100 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2013-07-23 21:36:18+0100 [tor] DEBUG: Retrying <GET https://check.torproject.org> (failed 1 times): 400 Bad Request
2013-07-23 21:36:18+0100 [tor] DEBUG: Retrying <GET https://check.torproject.org> (failed 2 times): 400 Bad Request
2013-07-23 21:36:18+0100 [tor] DEBUG: Gave up retrying <GET https://check.torproject.org> (failed 3 times): 400 Bad Request
2013-07-23 21:36:18+0100 [tor] DEBUG: Crawled (400) <GET https://check.torproject.org> (referer: None)
2013-07-23 21:36:18+0100 [tor] INFO: Closing spider (finished)

为了再次检查一下我的TOR/Polipo设置是否有问题，我可以在终端中运行以下curl命令，并且连接正常：curl --proxy localhost:8123 https://check.torproject.org/

对这里的问题有什么建议吗？

Answer 1

试一试

rq.meta['proxy'] = 'http://127.0.0.1:8123'

在我的情况下，它是有效的

Answer 2

不确定这些是否对您有帮助：

• How to use scrapy with an internet connection through a proxy with authentication
• http://mahmoud.abdel-fattah.net/2012/04/07/using-scrapy-with-proxies/
• Enabling HttpProxyMiddleware in scrapyd