无法使用美汤提取网页数据

Question

  url =  "https://www.telegraph.co.uk/formula-1/2018/08/25/f1-live-belgian-grand-prix-2018-qualifying-latest-updates/"
  soup = bs(urlopen(url), "lxml")
  divs = soup.findAll('div')
  base_url = "https://www.telegraph.co.uk"
  images = []
  print (divs)
  []

我得到的输出是空的。我认为这个页面是动态加载的。如何从该页面提取div。

Answer 1

页面内容由JS/动态加载，因此您必须使用selenium...you可以执行以下操作...

from bs4 import BeautifulSoup
from selenium import webdriver#you need to install selenium
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')
#copy your chromedriver to python folder
driver = webdriver.Chrome(chrome_options=options)
url =  ("https://www.telegraph.co.uk/"
"formula-1/2018/08/25/f1-live-belgian"
"-grand-prix-2018-qualifying-latest-updates/")
driver.get(url)
soup = BeautifulSoup(driver.page_source, 'lxml')
divs = soup.findAll('div')
print(divs)