向下滚动google地图网页

Question

我试图在谷歌地图网页上收集评论，但是我找不到一种方法来向下滚动页面来获得所有评论。

我使用的是python 3和Selenium包，

我已经找到了不同的方法来向下滚动社交媒体页面，如facebook或IG，但代码在谷歌地图上不起作用。另外，我试图定位body标签的结束键，但也不起作用。

如果有人能帮上忙我会很高兴，

先谢谢你，

Answer 1

您可以使用selenium中的execute_script滚动到页面底部。

browser = webdriver.Chrome('chromedriver.exe', options=self.browserProfile)
brownser.get('your url here')
browser.execute_script("window.scrollTo(0, document.body.scrollHeight);")

如果没有滚动到正确的区域，您可以在包含评论的目录上单击，然后执行window.scrollTo

 review_box= lambda: self.browser.find_element_by_xpath("xpath to div")
 review_box().click()

希望这能有所帮助！:)

Answer 2

使用selenium向下滚动的一种方法是单击页面上的特定元素。如果您执行以下操作...

reviews_divs = driver.find_elements_by_class_name('section-review')
reviews_divs[-1].click()

..。它将单击最后一次评论，因此滚动到它。很好，但还不是你想要的，因为页面没有加载新的结果……

但是，如果您单击section-review div之后的第一个div，新的评论现在可以正确加载：

driver.find_element_by_class_name('section-loading').click()

编辑

当你登陆你想要测试的商店页面时，点击“显示所有评论”后，上面的代码就会生效：

driver.find_element_by_css_selector("button[class*='__button-text'").click()

Answer 3

你也可以在没有浏览器自动化的情况下获得所有的评论。

您只需要data_id (它看起来像这样：0x89c259a61c75684f:0x79d31adb123348d2，您可以从一个地点的地图URL中获取它，或者在页面源中获取它)

​

​

在此之后，您只需向：https://www.google.com/async/reviewDialog?hl=en&async=feature_id:0x89c259a61c75684f:0x79d31adb123348d2,sort_by:,next_page_token:,associated_topic:,_fmt:pc发出请求

在那里您将找到所有的评论数据，以及next_page_token，因此您可以查询下10条评论。

在本例中，next_page_token为：EgIICg

因此，接下来10篇评论的请求URL将是：https://www.google.com/async/reviewDialog?hl=en&async=feature_id:0x89c259a61c75684f:0x79d31adb123348d2,sort_by:,next_page_token:EgIICg,associated_topic:,_fmt:pc

你也可以使用像SerpApi这样的第三方解决方案。这是一个免费试用的付费API。我们为您处理代理、解析验证码和解析所有丰富的结构化数据。

示例python代码(也可以在其他库中找到)：

from serpapi import GoogleSearch

params = {
  "api_key": "secret_api_key",
  "engine": "google_maps_reviews",
  "hl": "en",
  "data_id": "0x89c259a61c75684f:0x79d31adb123348d2"
}

search = GoogleSearch(params)
results = search.get_dict()

JSON输出示例：

"place_info": {
  "title": "Stumptown Coffee Roasters",
  "address": "18 W 29th St, New York, NY",
  "rating": 4.6,
  "reviews": 1343
},
"reviews": [
  {
    "user": {
      "name": "Julie Fowler",
      "link": "https://www.google.com/maps/contrib/117619864295803167167?hl=en-US&sa=X&ved=2ahUKEwj53uvVmuDxAhVkGFkFHYUJCDcQvvQBegQIARAy",
      "thumbnail": "https://lh3.googleusercontent.com/a-/AOh14GjzGpT8uNmp89FcD1OZ8nPouEOUwTLUJ4npewsY=s40-c-c0x00000000-cc-rp-mo-br100",
      "reviews": 4
    },
    "rating": 5,
    "date": "4 days ago",
    "snippet": "We popped by one day for a coffee, and the next day for a full-on breakfast. I loved the selection of coffee, teas, and non-alcoholic drinks. Friendly service topped off a great morning! They also have an excellent choice of coffee and a warm place to sit and visit. We will come back and recommend this place to my friends."
  },
  {
    "user": {
      "name": "Aida Penn",
      "link": "https://www.google.com/maps/contrib/100457332315069730904?hl=en-US&sa=X&ved=2ahUKEwj53uvVmuDxAhVkGFkFHYUJCDcQvvQBegQIARA-",
      "thumbnail": "https://lh3.googleusercontent.com/a-/AOh14GjfRec0fNDPIVR5F-68IFjJTLTGd_QYmXE7j5J8=s40-c-c0x00000000-cc-rp-mo-br100",
      "reviews": 1
    },
    "rating": 5,
    "date": "3 days ago",
    "snippet": "We had an early morning tour, and this was one of the only places open before 8 am. We went as soon as they opened to order cake and coffee to go. The staff were also very accommodating and nice even early in the morning. The coffee we got was good, but I wish we had time to go back and try more of the coffee here. I highly recommend this place!"
  },
  {
    "user": {
      "name": "Rhona Warren",
      "link": "https://www.google.com/maps/contrib/106431868819623834724?hl=en-US&sa=X&ved=2ahUKEwj53uvVmuDxAhVkGFkFHYUJCDcQvvQBegQIARBL",
      "thumbnail": "https://lh3.googleusercontent.com/a-/AOh14Gip2mBMR7-oIfh9z6JK1xr2O6SvDd-je_zFuZsq=s40-c-c0x00000000-cc-rp-mo-br100",
      "reviews": 1
    },
    "rating": 5,
    "date": "3 days ago",
    "snippet": "I stopped in for coffee then decided to order an espresso Oreo milkshake. Very delicious, and the prices were very reasonable. The staff was friendly, and there is nice outdoor patio seating. Also, the food was deliciously rich in flavor accompanied by terrific coffee and fresh fruit smoothies. I strongly recommend this place!"
  },
  ...
]

有关更多详细信息，请查看documentation。

免责声明:我在SerpApi工作。