BS4 -网络抓取-搜索div。类=

Question

我试图从下面的图像中抓取'95%喜欢这个电影‘，但是当我通过下面的类视图代码指定标签时，没有得到任何结果。知道我怎么能做到这一点吗？

import bs4, requests
from bs4 import BeautifulSoup
res = requests.get('https://www.google.com/search?rlz=1C5CHFA_enUS879US879&sxsrf=ALeKk00cw9xBpC8OWgCnKhMSIGOi4xb3sw%3A1590372307467&ei=0yfLXrSQHNHa9AOzh6jIAg&q=titanic+google+play&oq=Titanic+&gs_lcp=CgZwc3ktYWIQAxgAMgQIIxAnMgoIABCDARAUEIcCMgcIABCDARBDMgUIABCRAjIFCAAQkQIyBwgAEIMBEEMyBAgAEEMyBwgAEIMBEEMyBAgAEEMyBAgAEEM6BAgAEEc6AggAOgUIABCDAVCcLFjMOmCEQ2gBcAN4AIABbIgBigaSAQM4LjGYAQCgAQGqAQdnd3Mtd2l6&sclient=psy-ab')

res.raise_for_status()
soup = bs4.BeautifulSoup(res.text, 'html.parser')
news = [p.text.strip() for p in soup.select('h1 ~ p') if p.find('font')]

soup = BeautifulSoup(res.content, 'html.parser')
content = BeautifulSoup(res.content, 'html.parser')
content.find_all(class="srBp4.Vrkhme")`  

​

]1

Answer 1

谷歌使用JavaScript来显示结果-为了获得95% ...，我不得不使用Selenium来控制真正的web浏览器，后者可以运行JavaScript。我不得不使用查询titanic movie而不是titanic google play。

import selenium.webdriver

url = 'https://www.google.com/search?q=titanic+movie'

#driver = selenium.webdriver.Chrome()
driver = selenium.webdriver.Firefox()
driver.get(url)

item = driver.find_element_by_class_name('srBp4.Vrkhme')
print(item.text.strip())

编辑：--我也在requests/BeautifulSoup中获得了它，但是我不得不使用完整的标题User-Agent。它不适用于短Mozilla/5.0

它需要没有点的类"srBp4 Vrkhme"。它必须是class_=和_

import requests
from bs4 import BeautifulSoup

headers = {'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0'}
r = requests.get('https://www.google.com/search?q=titanic+movie', headers=headers)

soup = BeautifulSoup(r.content, 'html.parser')
item = soup.find('div', class_="srBp4 Vrkhme")
print(item.get_text(strip=True, separator=' '))