如何用Python抓取JavaScript页面

Question

我正在尝试抓取patentsview.org，但我遇到了一个问题。当我尝试抓取这个页面时，它不能很好地工作。使用JavaScript从其数据库中获取数据的站点。我试着用requests-html包获取数据，但我不太明白。

这是我尝试过的：

# Import
import re
from bs4 import BeautifulSoup
from requests_html import HTMLSession
session = HTMLSession()

# Set requests
r = session.get('https://datatool.patentsview.org/#search/assignee&asn=1|Samsung')
r.html.render()

# Set BS and print
soup = BeautifulSoup(r.html.html, "lxml")
tags = soup.find_all("div", class_='summary')
print(tags)

这段代码给出了如下结果：

# Result
[<div class="summary"></div>]

但我想要的是：

​

​

这就是正确的div。但是我的代码看不到div的内容。如何获取div的内容？希望你能理解我的意思。

Answer 1

使用浏览器开发工具。(Chrome。F12 - Network - XHR)并查看HTTP GET，它将返回您正在查找的数据(以JSON格式)。

HTTP GET https://webapi.patentsview.org/api/assignees/query?q={%22_and%22:[{%22_or%22:[{%22_and%22:[{%22_contains%22:{%22assignee_first_name%22:%22Samsung%22}}]},{%22_and%22:[{%22_contains%22:{%22assignee_last_name%22:%22Samsung%22}}]},{%22_and%22:[{%22_contains%22:{%22assignee_organization%22:%22Samsung%22}}]}]}]}&f=[%22assignee_id%22,%22assignee_first_name%22,%22assignee_last_name%22,%22assignee_organization%22,%22assignee_lastknown_country%22,%22assignee_lastknown_state%22,%22assignee_lastknown_city%22,%22assignee_lastknown_location_id%22,%22assignee_total_num_patents%22,%22assignee_first_seen_date%22,%22assignee_last_seen_date%22,%22patent_id%22]&o={%22per_page%22:50,%22matched_subentities_only%22:true,%22sort_by_subentity_counts%22:%22patent_id%22,%22page%22:1}&s=[{%22patent_id%22:%22desc%22},{%22assignee_total_num_patents%22:%22desc%22},{%22assignee_organization%22:%22asc%22},{%22assignee_last_name%22:%22asc%22},{%22assignee_first_name%22:%22asc%22}]