谷歌学者通过亚马逊网络服务ApiGateway抓取ip-rotator

Question

我收到下面的错误。代码(George方法，https://stackoverflow.com/users/7173479/george)在开始时工作了几次，但过了一段时间就崩溃了。这应该与HTTP的配置有关，但我在AWS文档中迷失了方向。我正在写jupyter笔记本。有人能帮上忙吗？

在AWS中创建网关对象并进行初始化

engine = 'https://scholar.google.com/scholar?hl=en&as_sdt=0%2C5&q={}&btnG='

gateway = ApiGateway(engine,\
                     access_key_id="KEY", access_key_secret="SECRET_KEY")
gateway.start()

将网关分配给会话

session = requests.Session()
session.mount(engine, gateway)

发送请求(IP将随机化)

header={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) \
    AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36'}

search_string = '{}+and+{}+and+{}+and+{}'.format('term1','term2','term3','term4')

url = engine.format(search_string)
print(url)

response = session.get(url,headers=header)
tree = BeautifulSoup(response.content,'lxml')
result = tree.find('div',id='gs_ab_md')

print(response.status_code)
print(result.text)
print(len(result.text))
number=[int(s.replace('.','').replace(',','')) for s in result.text.split() \
                if s.replace('.','').replace(',','').isdigit()]

删除网关

gateway.shutdown()

==

BadRequestException: An error occurred (BadRequestException) when calling the PutIntegration operation: Invalid HTTP endpoint specified for URI

Answer 1

requests-ip-rotator包中的ApiGateway构造函数的site参数应该只是站点。除了协议、域名或IP地址和端口之外，它不能包含URI的任何部分。

如果您将构造函数更改为如下所示：

gateway = ApiGateway("https://scholar.google.com")
gateway.start()

它将正确地构造网关端点。