网页抓取评论-Flipkart

Question

我正在尝试删除产品的整个评论(点击阅读更多后，显示评论的剩余一半。但我仍然不能做so.It没有显示评论的全部内容，这些内容在单击阅读更多选项后会显示出来。下面是代码，它单击readmore选项并从网站获取数据

import requests
from bs4 import BeautifulSoup
from selenium import webdriver

response = requests.get("https://www.flipkart.com/poco-f1-graphite-black-64-gb/product-reviews/itmf8fyjyssnt25c?page=2&pid=MOBF85V7A6PXETAX")    
data = BeautifulSoup(response.content, 'lxml')
chromepath = r"C:\Users\Mohammed\Downloads\chromedriver.exe"
driver=webdriver.Chrome(chromepath)
driver.get("https://www.flipkart.com/poco-f1-graphite-black-64-gb/product-reviews/itmf8fyjyssnt25c?page=2&pid=MOBF85V7A6PXETAX")
d = driver.find_element_by_class_name("_1EPkIx")
d.click()
title = data.find_all("p",{"class" : "_2xg6Ul"})
text1 = data.find_all("div",{"class" : "qwjRop"})
name = data.find_all("p",{"class" : "_3LYOAd _3sxSiS"})
for t2, t , t1 in zip(title,text1,name) :
    print(t2.text,'\n',t.text,'\n',t1.text)

Answer 1

要获得完整的评论，有必要单击这些READ MORE按钮来展开其余的内容。由于您已经将selenium与BeautifulSoup结合使用，因此我修改了脚本以符合逻辑。脚本将首先单击这些READ MORE按钮。一旦完成，它将从那里解析所有的titles和reviews。您现在可以从多个页面(最多4个页面)获取titles和reviews。

import time
from bs4 import BeautifulSoup
from selenium import webdriver

link = "https://www.flipkart.com/poco-f1-graphite-black-64-gb/product-reviews/itmf8fyjyssnt25c?page={}&pid=MOBF85V7A6PXETAX"

driver = webdriver.Chrome() #If necessary, define the chrome path explicitly
for page_num in range(1,5):
    driver.get(link.format(page_num))
    [item.click() for item in driver.find_elements_by_class_name("_1EPkIx")]
    time.sleep(1)
    soup = BeautifulSoup(driver.page_source, 'lxml')
    for items in soup.select("._3DCdKt"):
        title = items.select_one("p._2xg6Ul").text
        review = ' '.join(items.select_one(".qwjRop div:nth-of-type(2)").text.split())
        print(f'{title}\n{review}\n')

driver.quit()