从使用python和selenium的网页获取产品详细信息

Question

我需要从web页面中刮取的一些数据，我正在使用，但是没有人能够建议我应该做什么，或者如何修改代码以获得预期的结果，这将是非常有用的。

我需要得到以下订单位置，产品名称，URL和公司名称的结果。

下面的脚本可以得到所有的标题名称，但有额外的"\n“字符，不能得到其他的details.Can任何人请帮助我或对此提出建议？

from bs4 import BeautifulSoup
from selenium import webdriver 
from selenium.webdriver.common.by import By 
from selenium.webdriver.support.ui import WebDriverWait 
from selenium.webdriver.support import expected_conditions as EC 
from selenium.common.exceptions import TimeoutException

option = webdriver.ChromeOptions()
option.add_argument(" — incognito")
#browser = webdriver.Chrome(executable_path='/usr/local/bin/chromedriver/', chrome_options=option)
browser = webdriver.Chrome(executable_path='/users/user_1566/downloads/chrome_driver/chromedriver', chrome_options=option)

browser.get('https://www.google.com/search?q=samsung+note10')
#items = len(browser.find_elements_by_class_name("cu-container"))
#items = len(browser.find_elements_by_class_name("mnr-c pla-unit"))
#print(items)
timeout = 20
try:
    WebDriverWait(browser, timeout).until(EC.visibility_of_element_located((By.XPATH, "//div[@class='top-pla-group-inner']")))
except TimeoutException:
    print('Timed out waiting for page to load')
    #browser.quit()

titles_element = browser.find_elements_by_xpath("//div[@class='mnr-c pla-unit']")
# use list comprehension to get the actual repo titles and not the selenium objects.
titles = [x.text for x in titles_element]
# print out all the titles.
print('titles:')
print(titles, '\n')

language_element = browser.find_elements_by_xpath("//a[@class=''plantl pla-unit-single-clickable-target clickable-card']")
print(language_element)
# same concept as for list-comprehension above.
languages = [x.text for x in language_element]
print("languages:")
print(languages, "\n")


for title, language in zip(titles, languages):
    print("RepoName : Language")
    print(title + ": " + language, "\n")        

​

​

​

​

URL https://www.google.com/search?q=samsung+note10的预期输出

    Position  Company_Name     Product_Name                                 URL 

    1         CHECK24          Samsung Galaxy Note10 256 GB 
                               glow mit Vertrag + Allnet Flat bei                                                                     

Answer 1

试一试

details=browser.find_elements_by_class_name("pymv4e")
for i in details:
    print(str(i))

在那之后，你按你的意愿展示