用Python 3计算沃尔玛价格

Question

我对这个概念非常陌生，但我正在努力学习如何使用python来操作HTML数据。

我写了一条蟒蛇。3.4.1)获取URL并返回一些信息的脚本，我使用BeautifulSoup (ver )解析这些信息。4)。

import urllib.request
from bs4 import BeautifulSoup

response = urllib.request.urlopen('http://www.walmart.ca/en/ip/xbox-one/6000187109065')
html = response.read()

soup = BeautifulSoup(html)

print(soup.find_all('div', {"class" : "price-current"}))

在本例中，我试图获得Xbox One的价格。我选择这个div是因为它是在网页上向用户显示价格的那个。我知道有一个<span itemprop="price">$399.99</span>是可以刮的。从这是相当直截了当的，我更好奇的是，为什么价格不会出现在div，它应该出现。

我假设这与HTML标题有关，或者是某种POST/GET数据，在使用标准的web浏览器进行浏览时会自动发送。有谁能解释一下为什么价格没有出现，我要怎么做才能让他们如愿以偿呢？

Answer 1

价格使用javascript加载。这是请求：

Request URL:http://www.walmart.ca/ws/online/products
Request Method:POST

Form Data:
products:[{"productid":"6000187109066","skus":[[{"skuid":"6000187109066","status":"10"}]]}]
csrfToken:b08bfe580f3d9a0d893435fb

因为它包含一个csrfToken，所以在发出post请求之前，您需要了解它是如何生成或提供的。他们可能也依赖于会话cookie。

沃尔玛似乎有一个API：https://developer.walmartlabs.com/

Answer 2

免责声明:我在SerpApi工作。

您可以通过google-search-results包从SerpApi中节省沃尔玛产品的价格。这是一个免费试用的付费API。

全例

import os
from serpapi import GoogleSearch

params = dict(
    engine="walmart",
    query="xbox one",
    api_key=os.getenv("API_KEY"),
)

search = GoogleSearch(params)
results = search.get_dict()

for organic_result in results["organic_results"]:
    product_id = organic_result["us_item_id"] or organic_result["product_id"]

    search.params_dict = dict(
        engine="walmart_product",
        product_id=product_id,
        api_key=os.getenv("API_KEY"),
    )

    product_result = search.get_dict()

    print(
        f"Price of '{product_result['product_result']['title']}': '{product_result['product_result']['price_map']['price']} {product_result['product_result']['price_map']['currency']}'"
    )

输出

Price of 'Microsoft Xbox One S 1TB Console (Certified Refurbished)': '429 $'
Price of 'Star Wars Jedi: Fallen Order, Electronic Arts, Xbox One': '39.82 $'
Price of 'Microsoft - Xbox One S 1TB All-Digital Edition Console with Xbox One Wireless Controller (Renewed)': '499.99 $'
Price of 'Need for Speed: Heat, Electronic Arts, Xbox One, 014633373233': '25 $'
Price of 'Call of Duty: Black Ops Cold War, Activision, Xbox One': '49.94 $'
Price of 'NBA 2K21, 2K, Xbox One, 710425596858': '29.96 $'
Price of 'Refurbished Microsoft Factory Xbox One X 1TB, 4K Ultra HD Gaming Console in Black, FMQ-00042, 889842246971': '499.99 $'
Price of 'Madden NFL 21, Electronic Arts, Xbox One': '29.66 $'
Price of 'Need for Speed Payback, Electronic Arts, Xbox One, 014633370058': '14.96 $'
Price of 'Nickelodeon Kart Racers, Gamemill, Xbox One, 856131008060': '14.96 $'
Price of 'Sonic Mania (Sega), Xbox One, 010086640939': '14.96 $'
Price of 'Minecraft Master Collection, Microsoft, Xbox One, 889842394979': '42.88 $'
Price of 'Mortal Kombat 11, Warner Bros., Xbox One, 883929668977': '15 $'
Price of 'UFC 4, Electronic Arts, Xbox One': '29.96 $'
Price of 'Grand Theft Auto V, Rockstar Games, Xbox One': '22.8 $'
Price of 'Man Eater, Deep Silver, Xbox One, 816819017517': '24.96 $'
Price of 'Crash N. Sane Trilogy, Activision, Xbox One, 047875881969': '39.99 $'
Price of 'Spyro Reignited Trilogy, Activision, Xbox One, 047875882423': '29.83 $'
Price of 'Call of Duty: Modern Warfare, Activision, Xbox One, 0047875884366': '47.9 $'
Price of 'Cyberpunk 2077, Warner Bros, Xbox One': '39.74 $'
Price of 'Mass Effect Andromeda, Electronic Arts, Xbox One, 014633734096': '6.91 $'
Price of 'THQ theHunter: Call Of the Wild (Xbox One)': '19.93 $'
Price of 'Wreckfest, THQ-Nordic, Xbox One, 0811994021649': '29.83 $'
Price of 'Assassin’s Creed Valhalla Xbox Series X|S, Xbox One Standard Edition': '36.51 $'
Price of 'NBA 2K20, 2K, Xbox One, 710425595264': '21.99 $'
Price of 'LEGO Worlds, Warner Bros, Xbox One': '15 $'
Price of 'Call of Duty: Black Ops 3 Zombie Chronicles Edition, Activision, Xbox One, 047875881228': '37.88 $'
Price of 'Jumanji The Video Game, Bandai Namco, Xbox One, 819338020792': '19.93 $'
Price of 'Red Dead Redemption 2, Rockstar Games, Xbox One': '34.99 $'
Price of 'HALO 5, Microsoft, Xbox One, 885370928518': '19 $'
Price of 'LEGO Jurassic World, Warner, Xbox One, 883929472727': '19.8 $'
Price of 'Plants vs. Zombies: Battle for Neighborville, Electronic Arts, Xbox One, 014633736007': '14.96 $'
Price of 'The LEGO Marvel Collection, Warner Bros., Xbox One, 00883929670499': '24.98 $'
Price of 'LEGO The Incredibles, Warner Bros, Xbox One, 883929633005': '18 $'
Price of 'Assassin's Creed: Origins, Ubisoft, Xbox One, 887256028459': '19.93 $'
Price of 'Spongebob Squarepants: Battle for Bikini Bottom Rehydrated, THQ-Nordic, Xbox One, 811994022165': '29.77 $'
Price of 'The SIMS 4, Electronic Arts, Xbox One': '33.84 $'
Price of 'Star Wars Battlefront 2, Electronic Arts, Xbox One, 014633735321': '14.96 $'
Price of 'Pac-Man Championship Edition 2 + Arcade Game Series, Bandai/Namco, Xbox One, 722674220705': '14.49 $'
Price of 'Microsoft Xbox One S 1TB All Digital Edition 3 Game Bundle (Disc-free Gaming), White, NJP-00050': '198.95 $'