在请求中发现的刮伤和无效cookie

Question

Web抓取需求

要从eventbrite链接这里的第一页中抓取事件标题，请执行以下操作。

方法

虽然页面没有太多的javascript，页面分页很简单，但是获取页面上每个事件的标题是非常容易的，并且没有问题。

然而，我看到了一个API，我想重新设计HTTP请求，以提高效率和更多的结构化数据。

问题

我能够使用请求python包模拟HTTP请求，使用正确的标头、cookie和参数。不幸的是，当我使用与scrapy相同的cookie时，它似乎在抱怨cookie字典中的三个键，即空白的'mgrefby': ''、'ebEventToTrack': ''、'AN': ''。尽管它们在与请求包一起使用的HTTP请求中为空白。

请求包代码示例

import requests

cookies = {
    'mgrefby': '',
    'G': 'v%3D2%26i%3Dbff2ee97-9901-4a2c-b5b4-5189c912e418%26a%3Dd24%26s%3D7a302cadca91b63816f5fd4a0a3939f9c9f02a09',
    'ebEventToTrack': '',
    'eblang': 'lo%3Den_US%26la%3Den-us',
    'AN': '',
    'AS': '50c57c08-1f5b-4e62-8626-ea32b680fe5b',
    'mgref': 'typeins',
    'client_timezone': '%22Europe/London%22',
    'csrftoken': '85d167cac78111ea983bcbb527f01d2f',
    'SERVERID': 'djc9',
    'SS': 'AE3DLHRwcfsggc-Hgm7ssn3PGaQQPuCJ_g',
    'SP': 'AGQgbbkgEVyrPOfb8QOLk2Q893Bkx6aqepKtFsfXUC9SW6rLrY3HzVmFa6m91qZ6rtJdG0PEVaIXdCuyQOL27zgxTHS-Pn0nHcYFr9nb_gcU1ayxSx4Y0QXLDvhxGB9EMsou1MZmIfEBN7PKFp_enhYD6HUP80-pNUGLI9R9_CrpFzXc48lp8jXiHog_rTjy_CHSluFrXr2blZAJfdC8g2lFpc4KN8wtSyOwn8qTs7di3FUZAJ9BfoA',
}

headers = {
    'Connection': 'keep-alive',
    'User-Agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36',
    'X-Requested-With': 'XMLHttpRequest',
    'X-CSRFToken': '85d167cac78111ea983bcbb527f01d2f',
    'Content-Type': 'application/json',
    'Accept': '*/*',
    'Origin': 'https://www.eventbrite.com',
    'Sec-Fetch-Site': 'same-origin',
    'Sec-Fetch-Mode': 'cors',
    'Sec-Fetch-Dest': 'empty',
    'Referer': 'https://www.eventbrite.com/d/ny--new-york/human-resources/?page=2',
    'Accept-Language': 'en-US,en;q=0.9',
}

data = '{"event_search":{"q":"human resources","dates":"current_future","places":\n["85977539"],"page":1,"page_size":20,"online_events_only":false,"client_timezone":"Europe/London"},"expand.destination_event":["primary_venue","image","ticket_availability","saves","my_collections","event_sales_status"]}'

response = requests.post('https://www.eventbrite.com/api/v3/destination/search/', headers=headers, cookies=cookies, data=data)

刮除代码示例

class TestSpider(scrapy.Spider):
    name = 'test'
    allowed_domains = ['eventbrite.com']
    start_urls = []

    cookies = {
    'mgrefby': '',
    'G': 'v%3D2%26i%3Dbff2ee97-9901-4a2c-b5b4-5189c912e418%26a%3Dd24%26s%3D7a302cadca91b63816f5fd4a0a3939f9c9f02a09',
    'ebEventToTrack': '',
    'eblang': 'lo%3Den_US%26la%3Den-us',
    'AN': '',
    'AS': '50c57c08-1f5b-4e62-8626-ea32b680fe5b',
    'mgref': 'typeins',
    'client_timezone': '%22Europe/London%22',
    'csrftoken': '85d167cac78111ea983bcbb527f01d2f',
    'SERVERID': 'djc9',
    'SS': 'AE3DLHRwcfsggc-Hgm7ssn3PGaQQPuCJ_g',
    'SP': 'AGQgbbkgEVyrPOfb8QOLk2Q893Bkx6aqepKtFsfXUC9SW6rLrY3HzVmFa6m91qZ6rtJdG0PEVaIXdCuyQOL27zgxTHS-Pn0nHcYFr9nb_gcU1ayxSx4Y0QXLDvhxGB9EMsou1MZmIfEBN7PKFp_enhYD6HUP80-pNUGLI9R9_CrpFzXc48lp8jXiHog_rTjy_CHSluFrXr2blZAJfdC8g2lFpc4KN8wtSyOwn8qTs7di3FUZAJ9BfoA',
}

    headers = {
        'Connection': 'keep-alive',
        'User-Agent': 'Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Mobile Safari/537.36',
        'X-Requested-With': 'XMLHttpRequest',
        'X-CSRFToken': '85d167cac78111ea983bcbb527f01d2f',
        'Content-Type': 'application/json',
        'Accept': '*/*',
        'Origin': 'https://www.eventbrite.com',
        'Sec-Fetch-Site': 'same-origin',
        'Sec-Fetch-Mode': 'cors',
        'Sec-Fetch-Dest': 'empty',
        'Referer': 'https://www.eventbrite.com/d/ny--new-york/human-resources/?page=1',
        'Accept-Language': 'en-US,en;q=0.9',
    }

    data = '{"event_search":{"q":"human resources","dates":"current_future","places":\n["85977539"],"page":1,"page_size":20,"online_events_only":false,"client_timezone":"Europe/London"},"expand.destination_event":["primary_venue","image","ticket_availability","saves","my_collections","event_sales_status"]}'


    def start_requests(self):
        url = 'https://www.eventbrite.com/api/v3/destination/search/'
        yield scrapy.Request(url=url, method='POST',headers=self.headers,cookies=self.cookies,callback=self.parse)
    def parse(self,response):
        print('request')

输出

2020-08-01 11:55:33 [scrapy.core.engine] INFO: Spider opened
2020-08-01 11:55:33 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-08-01 11:55:33 [test] INFO: Spider opened: test
2020-08-01 11:55:33 [scrapy.extensions.httpcache] DEBUG: Using filesystem cache storage in C:\Users\Aaron\projects\scrapy\eventbrite\.scrapy\httpcache
2020-08-01 11:55:33 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-08-01 11:55:33 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.eventbrite.com/robots.txt> (referer: None) ['cached']
2020-08-01 11:55:33 [scrapy.downloadermiddlewares.cookies] WARNING: Invalid cookie found in request <POST https://www.eventbrite.com/api/v3/destination/search/>: {'name': 'mgrefby', 'value': ''} ('value' is missing)
2020-08-01 11:55:33 [scrapy.downloadermiddlewares.cookies] WARNING: Invalid cookie found in request <POST https://www.eventbrite.com/api/v3/destination/search/>: {'name': 'ebEventToTrack', 'value': ''} ('value' is missing)
2020-08-01 11:55:33 [scrapy.downloadermiddlewares.cookies] WARNING: Invalid cookie found in request <POST https://www.eventbrite.com/api/v3/destination/search/>: {'name': 'AN', 'value': ''} ('value' is missing)   
2020-08-01 11:55:33 [scrapy.core.engine] DEBUG: Crawled (401) <POST https://www.eventbrite.com/api/v3/destination/search/> (referer: https://www.eventbrite.com/d/ny--new-york/human-resources/?page=1) ['cached']   
2020-08-01 11:55:33 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <401 https://www.eventbrite.com/api/v3/destination/search/>: HTTP status code is not handled or not allowed
2020-08-01 11:55:33 [scrapy.core.engine] INFO: Closing spider (finished)
2020-08-01 11:55:33 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 1540,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 1,
 'downloader/request_method_count/POST': 1,
 'downloader/response_bytes': 32163,
 'downloader/response_count': 2,
 'downloader/response_status_count/200': 1,
 'downloader/response_status_count/401': 1,
 'elapsed_time_seconds': 0.187986,
 'finish_reason': 'finished',
 'finish_time': datetime.datetime(2020, 8, 1, 10, 55, 33, 202931),
 'httpcache/hit': 2,
 'httperror/response_ignored_count': 1,
 'httperror/response_ignored_status_count/401': 1,
 'log_count/DEBUG': 3,
 'log_count/INFO': 12,
 'log_count/WARNING': 3,
 'response_received_count': 2,
 'robotstxt/request_count': 1,
 'robotstxt/response_count': 1,
 'robotstxt/response_status_count/200': 1,
 'scheduler/dequeued': 1,
 'scheduler/dequeued/memory': 1,
 'scheduler/enqueued': 1,
 'scheduler/enqueued/memory': 1,
 'start_time': datetime.datetime(2020, 8, 1, 10, 55, 33, 14945)}
2020-08-01 11:55:33 [scrapy.core.engine] INFO: Spider closed (finished)

解决问题的尝试

401状态似乎是指授权，我只能推测它不喜欢我发送的饼干。

1. 我将COOKIES_ENABLED = True设置为与前面相同的输出
2. 我已经设置了COOKIES_DEBUG = True，并看到了下面的输出

用cookies_debug=True输出

2020-08-01 12:05:15 [scrapy.core.engine] INFO: Spider opened
2020-08-01 12:05:15 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2020-08-01 12:05:15 [test] INFO: Spider opened: test
2020-08-01 12:05:15 [scrapy.extensions.httpcache] DEBUG: Using filesystem cache storage in C:\Users\Aaron\projects\scrapy\eventbrite\.scrapy\httpcache
2020-08-01 12:05:15 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2020-08-01 12:05:15 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://www.eventbrite.com/robots.txt> (referer: None) ['cached']
2020-08-01 12:05:15 [scrapy.downloadermiddlewares.cookies] WARNING: Invalid cookie found in request <POST https://www.eventbrite.com/api/v3/destination/search/>: {'name': 'mgrefby', 'value': ''} ('value' is missing)
2020-08-01 12:05:15 [scrapy.downloadermiddlewares.cookies] WARNING: Invalid cookie found in request <POST https://www.eventbrite.com/api/v3/destination/search/>: {'name': 'ebEventToTrack', 'value': ''} ('value' is missing)
2020-08-01 12:05:15 [scrapy.downloadermiddlewares.cookies] WARNING: Invalid cookie found in request <POST https://www.eventbrite.com/api/v3/destination/search/>: {'name': 'AN', 'value': ''} ('value' is missing)   
2020-08-01 12:05:15 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <POST https://www.eventbrite.com/api/v3/destination/search/>
Cookie: G=v%3D2%26i%3Dbff2ee97-9901-4a2c-b5b4-5189c912e418%26a%3Dd24%26s%3D7a302cadca91b63816f5fd4a0a3939f9c9f02a09; eblang=lo%3Den_US%26la%3Den-us; AS=50c57c08-1f5b-4e62-8626-ea32b680fe5b; mgref=typeins; client_timezone=%22Europe/London%22; csrftoken=85d167cac78111ea983bcbb527f01d2f; SERVERID=djc9; SS=AE3DLHRwcfsggc-Hgm7ssn3PGaQQPuCJ_g; SP=AGQgbbkgEVyrPOfb8QOLk2Q893Bkx6aqepKtFsfXUC9SW6rLrY3HzVmFa6m91qZ6rtJdG0PEVaIXdCuyQOL27zgxTHS-Pn0nHcYFr9nb_gcU1ayxSx4Y0QXLDvhxGB9EMsou1MZmIfEBN7PKFp_enhYD6HUP80-pNUGLI9R9_CrpFzXc48lp8jXiHog_rTjy_CHSluFrXr2blZAJfdC8g2lFpc4KN8wtSyOwn8qTs7di3FUZAJ9BfoA

2020-08-01 12:05:15 [scrapy.downloadermiddlewares.cookies] DEBUG: Received cookies from: <401 https://www.eventbrite.com/api/v3/destination/search/>
Set-Cookie: SP=AGQgbbno_KHLNiLzDpLHcdI4kotUbRiTxMMY5N0t7VudPU_QGCm2Q0nH7-J99aoRZvGLxXfREH5YfPAtK52iiiLcEpnjh1G43ZBxKuo9qvJHykLV23ZIjaFK0iIr6ptOaczMoQhkaqE-7nJ8t2Ykt18CN196pKZ5QhFuXy6SnspZ0toEGChZcQgmrAAAVPfuoiiUmbTG_wJC8_KikL2sYl2s6-KWUOOpjRFJCko5RGgiyC2Osu9vxZ8; Domain=.eventbrite.com; httponly; Path=/; secure

Set-Cookie: G=v%3D2%26i%3D5cebebd2-2a7f-4638-9912-0abf19111a0c%26a%3Dd33%26s%3Df967e32d15dda2f06b392f22451af935d93f88d1; Domain=.eventbrite.com; expires=Sat, 31-Jul-2021 22:46:28 GMT; httponly; Path=/; secure     

Set-Cookie: ebEventToTrack=; Domain=.eventbrite.com; expires=Sun, 30-Aug-2020 22:46:28 GMT; httponly; Path=/; secure

Set-Cookie: SS=AE3DLHRgTIL46n9XiOZiJRSkccGnNXSMkA; Domain=.eventbrite.com; httponly; Path=/; secure

Set-Cookie: eblang=lo%3Den_US%26la%3Den-us; Domain=.eventbrite.com; expires=Sat, 31-Jul-2021 22:46:28 GMT; httponly; Path=/; secure

Set-Cookie: AN=; Domain=.eventbrite.com; expires=Sun, 30-Aug-2020 22:46:28 GMT; httponly; Path=/; secure

Set-Cookie: AS=350def0c-ed27-45ab-b12c-02e9fb68a8ae; Domain=.eventbrite.com; httponly; Path=/; secure

Set-Cookie: SERVERID=djc44; path=/; HttpOnly; Secure

2020-08-01 12:05:15 [scrapy.core.engine] DEBUG: Crawled (401) <POST https://www.eventbrite.com/api/v3/destination/search/> (referer: https://www.eventbrite.com/d/ny--new-york/human-resources/?page=1) ['cached']
2020-08-01 12:05:15 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <401 https://www.eventbrite.com/api/v3/destination/search/>: HTTP status code is not handled or not allowed
2020-08-01 12:05:15 [scrapy.core.engine] INFO: Closing spider (finished)

1. 我尝试了一个用于cookies持久性的自定义cookies下载器中间件，并再次犯了与以前相同的错误。
2. 我已经考虑过使用浏览器自动化来抓取一个cookie，再一次想到这一点，就像在将来我不想继续抓取一个cookie的情况一样。

我不明白的是，对于请求python包中相同的cookie、标头和参数，存在JSON对象响应。有了刮痕，它在抱怨空白的字典值。

如果我犯了一个明显的错误，或者想知道为什么API端点通过请求接受的cookie在Scrapy中似乎不起作用，我将不胜感激。

Answer 1

看起来他们使用的是not value而不是更精确的value is not None。开始发行是您唯一的长期追索权，但是对cookie中间件进行子类化是一个短期的、无恶意的解决方案。

恶意修复是利用这样一个事实:在执行'; '.join()时，它们没有正确地转义cookie值，因此您可以将cookie的值设置为合法的cookie指令(我选择了HttpOnly，因为您不关心JS)，而cookiejar似乎放弃了它，产生了您关心的实际值。

>>> from scrapy.downloadermiddlewares.cookies import CookiesMiddleware
>>> from scrapy.http import Request
>>> cm = CookiesMiddleware(debug=True)
>>> req = Request(url='https://www.example.com', cookies={'AN': '; HttpOnly', 'alpha': 'beta'})
>>> cm.process_request(req, spider=None)
2020-08-01 15:08:58 [scrapy.downloadermiddlewares.cookies] DEBUG: Sending cookies to: <GET https://www.example.com>
Cookie: AN=; alpha=beta
>>> req.headers
{b'Cookie': [b'AN=; alpha=beta']}

Answer 2

添加到Mdaniel的响应中，我已经打开了一个问题，因为我们遇到了同样的问题，并且引用了您的堆栈溢出线程。

我们目前的解决方案是使用旧版本的scrapy (2.2.0或更低版本)，因为最新的2.3.0是添加此cookie检查的地方。https://github.com/scrapy/scrapy/commit/f6ed5edc31e7cc66225c0860e1534a6230511954 scrapy/下载中间件/炊具. 78第78行

如果你想补充我遗漏的任何东西，这就是问题所在。https://github.com/scrapy/scrapy/issues/4766