Web报废:为什么我的代码不输出所需的输出？

Question

我正在使用python语言和BeautifulSoup进行网络抓取项目。我尝试了下面的示例代码并执行所需的输出。

from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as uReq

URL = "https://www.newegg.com/Monitors/Category/ID-19?Tpk=monitor"
uClient = uReq(URL)
page_html = uClient.read()

page_soup = soup(page_html, "html.parser")
uClient.close()

containers = page_soup.findAll("div", {"class": "item-container"})
print(containers)

但是，当我调整代码(链接和html标记部分)时，输出是[]。是因为链接还是因为html标记？

from bs4 import BeautifulSoup as soup
from urllib.request import urlopen as uReq

URL = "https://www.lotuss.com.my/en/category/grocery"
uClient = uReq(URL)
page_html = uClient.read()

page_soup = soup(page_html, "html.parser")
uClient.close()

containers = page_soup.findAll("div",{"class":"product-grid-item"})
print(containers)

Answer 1

问题似乎是在您提出的请求中没有返回product-grid-item。看起来他们正在使用react，所以它很可能是生成客户端的，而您的所有请求都是作为react的根的index.html文件获得的。

最好的选择是使用类似Selenium：https://theleakycauldronblog.com/blog/scraping-dynamic-single-page-websites-with-scrapy-and-selenium之类的东西

Answer 2

该项目是动态接收的，您无法通过bs4获得它。您必须找到一个将返回所需项的查询。在研究了请求结构之后，您将面临另一个问题，即如何获得访问令牌。简单：

def get_token():
    url = "https://identity-service-online-ecommerce-my-prod.auth.ap-southeast-1.amazoncognito.com/oauth2/token"
    payload = 'grant_type=client_credentials&scope=service-api%2Fgeneral'
    headers = {
      'authority': 'identity-service-online-ecommerce-my-prod.auth.ap-southeast-1.amazoncognito.com',
      'accept': 'application/json, text/plain, */*',
      'authorization': 'Basic M200YjE4bDhvZ3NhM2s0dXV0cDFxZ3VuZTg6MW4xaXJkMGJoZ2dzbGZlMTFiM3NsYWl0YjVyN3BjZG4xcDhqYWJsNHZiaGJtZTRibHVmYQ==',
      'content-type': 'application/x-www-form-urlencoded',
      'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
    }

    response = requests.request("POST", url, headers=headers, data=payload)
    return response.json()['access_token']

现在我们可以将必要的参数放入函数中，首先想到的是抵消、标记和我们正在寻找产品的类别。分页产品需要偏移量。由于我查看了您所附的url，我们对Grocery感兴趣，它的ID为2730。例如，我将显示90种产品，在每个产品中指定sku、名称、价格和url。完整代码：

import requests
import json


def get_token():
    url = "https://identity-service-online-ecommerce-my-prod.auth.ap-southeast-1.amazoncognito.com/oauth2/token"
    payload = 'grant_type=client_credentials&scope=service-api%2Fgeneral'
    headers = {
      'authority': 'identity-service-online-ecommerce-my-prod.auth.ap-southeast-1.amazoncognito.com',
      'accept': 'application/json, text/plain, */*',
      'authorization': 'Basic M200YjE4bDhvZ3NhM2s0dXV0cDFxZ3VuZTg6MW4xaXJkMGJoZ2dzbGZlMTFiM3NsYWl0YjVyN3BjZG4xcDhqYWJsNHZiaGJtZTRibHVmYQ==',
      'content-type': 'application/x-www-form-urlencoded',
      'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36',
    }

    response = requests.request("POST", url, headers=headers, data=payload)
    return response.json()['access_token']


def show_items(token, offset, category_id):
    url = "https://shoponline-bffapi.lotuss.com.my/bff/product/v1/products?websiteCode=malaysia_hy"

    payload = json.dumps({
        "offset": offset,
        "limit": 30,
        "filter": {
            "categoryId": category_id
        }
    })
    headers = {
        'accept': 'application/json, text/plain, */*',
        'accept-language': 'en',
        'authorization': token,
        'content-type': 'application/json',
        'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36'
    }
    response = requests.request("POST", url, headers=headers, data=payload)
    for product in response.json()['data']['products']:
        price = str(product['priceRange']['minimumPrice']['finalPrice']['value']) + product['priceRange']['minimumPrice']['finalPrice']['currency']
        url = 'https://www.lotuss.com.my/en/product/' + product['urlKey']
        print(product['sku'], product['name'], price, url)


access_token = get_token()
for page_offset in range(30, 120, 30):
    show_items(access_token, page_offset, ["2730"])

产出：

74697196 LOTUSS GREEN PEA SNACK 60G 1.09MYR https://www.lotuss.com.my/en/product/lotuss-green-pea-snack-60g-74697196
79170 SAN REMO NO.5 SPAGHETTI 500G 4.95MYR https://www.lotuss.com.my/en/product/san-remo-no-5-spaghetti-500g-79170
510637 CHACHEER SUNFLOWER SEEDS 220G 5.99MYR https://www.lotuss.com.my/en/product/chacheer-sunflower-seeds-220g-510637
2729881 TAO KAE NOI CRISPY SEAWEED ORIG 32.5G 6.59MYR https://www.lotuss.com.my/en/product/tao-kae-noi-crispy-seaweed-orig-32-5g-2729881
71138730 POCKY CHOCOLATE FLAVOUR 40G 2.95MYR https://www.lotuss.com.my/en/product/pocky-chocolate-flavour-40g-71138730
72527811 JACOBS CREAM CRACKER 700G 13.5MYR https://www.lotuss.com.my/en/product/jacobs-cream-cracker-700g-72527811
72338458 CED NATURAL HIMALAYAN PINK ROCK SALT 2.29MYR https://www.lotuss.com.my/en/product/ced-natural-himalayan-pink-rock-salt-72338458
74698990 Lotus's Ridge Cut Original Potato Chips 70g 2.19MYR https://www.lotuss.com.my/en/product/lotuss-potato-chips-r-cut-70g-original-74698990
243108 COCON PUDDING LYCHEE + FRIENDS 12X80G 11.99MYR https://www.lotuss.com.my/en/product/cocon-pudding-lychee-+-friends-12x80g-243108
1363964 CED ORGANIC CANE SUGAR 500G 5.59MYR https://www.lotuss.com.my/en/product/ced-organic-cane-sugar-500g-1363964
3788679 E.S BRAND PICKLED LETTUCE 182G 2.99MYR https://www.lotuss.com.my/en/product/e-s-brand-pickled-lettuce-182g-3788679
2220679 SUPER RING CHEESE FAMILY PACK 8X14G 2.79MYR https://www.lotuss.com.my/en/product/super-ring-cheese-family-pack-8x14g-2220679
3341844 MUNCHYS OATKRUNCH DARK CHOC 390G 7.99MYR https://www.lotuss.com.my/en/product/munchys-oatkrunch-dark-choc-390g-3341844
3267024 FRASER HILLS HONEY 350G 6.71MYR https://www.lotuss.com.my/en/product/fraser-hills-honey-350g-3267024
3263746 BERASLI INSTANT PORRIDGE CHICKEN 40G 1.95MYR https://www.lotuss.com.my/en/product/berasli-instant-porridge-chicken-flavoured-40g-3263746
3242846 SNEKKU FAMILY PACK-TAM TAM 8X22G 2.99MYR https://www.lotuss.com.my/en/product/snekku-family-pack-tam-tam-8x25g-3242846
74138936 PRAI COARSE GRAIN SUGAR 1KG 2.85MYR https://www.lotuss.com.my/en/product/prai-coarse-grain-sugar-1kg-74138936
74690248 LOTUSS BUNGA CENGKIH 50G 4.69MYR https://www.lotuss.com.my/en/product/lotuss-bunga-cengkih-50g-74690248
1364901 A1 SOUP SPICES 35G 8.75MYR https://www.lotuss.com.my/en/product/a1-bak-kut-teh-soup-spices-35g-1364901
557285 KARA COCONUT CREAM 200ML 3.19MYR https://www.lotuss.com.my/en/product/kara-coconut-cream-200ml-557285
125733 NESTLE KOKO KRUNCH ECONOPACK 500G 14.99MYR https://www.lotuss.com.my/en/product/nestle-koko-krunch-econopack-500g-125733
72338474 CED ORG REGULAR ROLLED OATS 500G 7.99MYR https://www.lotuss.com.my/en/product/ced-org-regular-rolled-oats-500g-72338474
70655871 ADELA TUB  MARGARINE 1 KG 11.09MYR https://www.lotuss.com.my/en/product/adela-multi-purpose-margarine-1kg-70655871
4638255 JULIES OAT 25 TEN GRAINS 300G 8.05MYR https://www.lotuss.com.my/en/product/julies-oat-25-ten-grains-300g-4638255
4411757 MAGGI 2MIN CHICKEN FLAVOR 5X77G 3.99MYR https://www.lotuss.com.my/en/product/maggi-2min-chicken-flavor-5x77g-4411757
4350197 DAISY PEANUT BUTTER (CRUNCHY) 500G 11.79MYR https://www.lotuss.com.my/en/product/daisy-peanut-butter-crunchy-500g-4350197
2959216 SMUCKERS SUGAR FREE STRAWBERRY 361G 11.59MYR https://www.lotuss.com.my/en/product/smuckers-sugar-free-strawberry-361g-2959216
2252651 MENTOS MINT POUCH 36S (97.2G) 2.79MYR https://www.lotuss.com.my/en/product/mentos-mint-pouch-36s-97-2g-2252651
2605791 TUMIX CHICKEN FLAVOR 1KG 17.99MYR https://www.lotuss.com.my/en/product/tumix-chicken-flavor-1kg-2605791
897892 DOUBLE SWALLOW BARLI 100G 0.79MYR https://www.lotuss.com.my/en/product/double-swallow-barli-100g-897892
116122 ADABI SERBUK BIHUN SUP 20G 1.49MYR https://www.lotuss.com.my/en/product/adabi-serbuk-bihun-sup-20g-116122
553700 TAMIN TAUCU 450G 3.64MYR https://www.lotuss.com.my/en/product/tamin-taucu-450g-553700
512176 VOCHELLE DOYPK ASSORTMENT 100G 6.39MYR https://www.lotuss.com.my/en/product/vochelle-doypk-assortment-100g-512176
412260 YEOS BAKED BEAN IN TMT SOS 425G 4.45MYR https://www.lotuss.com.my/en/product/yeos-baked-bean-in-tmt-sos-425g-412260
73925446 LE DRIED MULBERRY 140G 13.99MYR https://www.lotuss.com.my/en/product/le-dried-mulberry-140g-73925446
74690582 LOTUSS BLACK RAISIN 120G 3.99MYR https://www.lotuss.com.my/en/product/lotuss-black-raisin-120g-74690582
414700 ROTA SUPER RING 60G 1.99MYR https://www.lotuss.com.my/en/product/rota-super-ring-60g-414700
1446169 BOP PHKARKHNEI NATURAL WHITE RICE 5KG 38.79MYR https://www.lotuss.com.my/en/product/bop-phkarkhnei-natural-white-rice-5kg-1446169
4358996 KNORR CUP PORRIDGE CHIC+MUSH 35G 2.9MYR https://www.lotuss.com.my/en/product/knorr-cup-porridge-chic+mush-35g-4358996
72830735 CAP LILI FORTIFIED WHEAT FLOUR 850G 2.35MYR https://www.lotuss.com.my/en/product/cap-lili-fortified-wheat-flour-850g-72830735
74700308 LOTUSS CHIPS 110G SOUR CREAM+ONION 2.49MYR https://www.lotuss.com.my/en/product/lotuss-chips-110g-sour-cream+onion-74700308
74696009 LOTUSS BLACK RAISIN 250G POUCH 7.99MYR https://www.lotuss.com.my/en/product/lotuss-black-raisin-250g-pouch-74696009
70856206 NESTUM 3-IN-1 CEREAL ORIGINAL 15X28G 9.99MYR https://www.lotuss.com.my/en/product/nestum-3-in-1-cereal-original-15x28g-70856206
74377485 SWISS LEF SWISS ROLL STRAWBERRY 6S 20G 2.79MYR https://www.lotuss.com.my/en/product/swiss-lef-swiss-roll-strawberry-6s-20g-74377485
74283677 COCON FRUIT CUBE PUDDING MANGO 118GX6 8.99MYR https://www.lotuss.com.my/en/product/cocon-fruit-cube-pudding-mango-118gx6-74283677
74547321 LE ORG EDAMAME FETTUCCINE 200G 12.99MYR https://www.lotuss.com.my/en/product/le-org-edamame-fettuccine-200g-74547321
74689584 LOTUSS BARLI 300G 2.49MYR https://www.lotuss.com.my/en/product/lotuss-barli-300g-74689584
120987 JACOBS WEETAMEAL 700G 13.5MYR https://www.lotuss.com.my/en/product/jacobs-weetameal-700g-120987
410675 KING CUP SARDINE 155G 4.19MYR https://www.lotuss.com.my/en/product/king-cup-sardine-155g-410675
410543 FAIZA EMAS SST 5% 10KG 24.29MYR https://www.lotuss.com.my/en/product/faiza-emas-sst-5-10kg-410543
3143996 YEOS CURRY CHICKEN 145G 4.95MYR https://www.lotuss.com.my/en/product/yeos-curry-chicken-145g-3143996
70323232 ARM / HAMMER PURE BAKING SODA 454G 7.29MYR https://www.lotuss.com.my/en/product/arm-&-hammer-pure-baking-soda-454g-70323232
72866039 B CROCKER PANCKE MIX BUTTERMILK 430G 9.99MYR https://www.lotuss.com.my/en/product/b-crocker-pancke-mix-buttermilk-430g-72866039
71428852 AH FOOK HERBS + SPICES MIXED 35G 5.99MYR https://www.lotuss.com.my/en/product/ah-fook-klang-herbs-&-spices-mixed-35g-71428852
73925624 LE CHILLI POWDER 60G 9.99MYR https://www.lotuss.com.my/en/product/le-chilli-powder-60g-73925624
72318899 CAMERON GARDEN BUTTON MUSHROOM CAN 425G 3.4MYR https://www.lotuss.com.my/en/product/cameron-garden-button-mushroom-can-425g-72318899
73785075 CED ORGANIC MOLASSES SUGAR 850G 13.99MYR https://www.lotuss.com.my/en/product/ced-organic-molasses-sugar-850g-73785075
74690175 LOTUSS COARSE SUGAR 1KG 2.85MYR https://www.lotuss.com.my/en/product/lotuss-coarse-sugar-1kg-74690175
74689649 LOTUSS KAYU MANIS 50G 3.79MYR https://www.lotuss.com.my/en/product/lotuss-kayu-manis-50g-74689649
74682237 LOTUSS SPAGHETTI 400G 2.49MYR https://www.lotuss.com.my/en/product/lotuss-spaghetti-400g-74682237
2544776 CAMPBELL CREAM OF MUSHROOM 3X21.1G 4.75MYR https://www.lotuss.com.my/en/product/campbell-cream-of-mushroom-3x21-1g-2544776
243183 COCON YOGO ICE COMBO PCK 10X45ML 5.09MYR https://www.lotuss.com.my/en/product/cocon-yogo-ice-combo-pck-10x45ml-243183
74738887 OHALO ORG COCONUT ALMOND SPREAD 135G 15.59MYR https://www.lotuss.com.my/en/product/ohalo-org-coconut-almond-spread-135g-74738887
2609347 JASMINE BEEHOON 400G 2.65MYR https://www.lotuss.com.my/en/product/jasmine-beehoon-400g-2609347
72486384 MUNCHYS OAT KRUNCH NUTTY CHOCOLATE 390G 7.99MYR https://www.lotuss.com.my/en/product/munchys-oat-krunch-nutty-chocolate-390g-72486384
71138722 POCKY STRAWBERRY FLAVOUR 38G 2.95MYR https://www.lotuss.com.my/en/product/pocky-strawberry-flavour-38g-71138722
74706527 LOTUSS MURUKU 200G PAKODA 3.99MYR https://www.lotuss.com.my/en/product/lotuss-muruku-200g-pakoda-74706527
74690426 LOTUSS CENDAWAN CINA 45G 5.49MYR https://www.lotuss.com.my/en/product/lotuss-cendawan-cina-45g-74690426
74557386 SAJI SANTAN KELAPA 200ML 2.9MYR https://www.lotuss.com.my/en/product/saji-santan-kelapa-200ml-74557386
74554093 LE COCONUT SYRUP 270G 13.99MYR https://www.lotuss.com.my/en/product/le-coconut-syrup-270g-74554093
74698869 LOTUSS HONEY 1KG 12.99MYR https://www.lotuss.com.my/en/product/lotuss-honey-1kg-74698869
4115252 TAO KAE NOI BIG BANG CLASSIC 50G 10.19MYR https://www.lotuss.com.my/en/product/tao-kae-noi-big-bang-classic-50g-4115252
71677100 MAMEE MONSTER  HOT+SPICY 8X25G 3MYR https://www.lotuss.com.my/en/product/mamee-monster-family-pack-h+s-8x25g-71677100
3806049 PENGUIN LYCHEE 567G 5.49MYR https://www.lotuss.com.my/en/product/penguin-lychee-567g-3806049
74698966 Lotus's Hot & Spicy Flavoured Potato Chips 70g 2.19MYR https://www.lotuss.com.my/en/product/lotuss-potato-chips-f-cut-70g-hot-spicy-74698966
74698435 LOTUSS CHICKEN STOCK CUBE 6X11G 2.89MYR https://www.lotuss.com.my/en/product/lotuss-chicken-stock-cube-6x11g-74698435
1041584 TESCO PITTED BLACK OLIVES 330G(D/W 163G) 6.21MYR https://www.lotuss.com.my/en/product/tesco-pitted-black-olives-330g-d-w-163g-1041584
378860 STAR BRAND FLAVOUR VANILLA 25ML 2.39MYR https://www.lotuss.com.my/en/product/star-brand-flavour-vanilla-25ml-378860
71002162 JIF PEANUT BUTTER - CRUNCHY 16OZ 16.29MYR https://www.lotuss.com.my/en/product/jif-peanut-butter-crunchy-16oz-71002162
3245926 YUPI STRAWBERRY KISS 120G 4.15MYR https://www.lotuss.com.my/en/product/yupi-strawberry-kiss-120g-3245926
2959399 SMUCKERS SUGAR FREE CONCORDE GRAPE 361G 11.59MYR https://www.lotuss.com.my/en/product/smuckers-sugar-free-concorde-grape-361g-2959399
117536 ADABI PERENCAH SUP SIAM 40G 1.35MYR https://www.lotuss.com.my/en/product/adabi-thai-soup-paste-40g-117536
518905 KOKO JELLY NUT 380G 7.59MYR https://www.lotuss.com.my/en/product/koko-jelly-nut-380g-518905
73973319 CFO RAISIN PROMO PACK 300GX2 25.9MYR https://www.lotuss.com.my/en/product/cfo-raisin-promo-pack-300gx2-73973319
74697234 LOTUSS CHEESE FLAVOUR RINGS 60G 1.09MYR https://www.lotuss.com.my/en/product/lotuss-cheese-flavour-rings-60g-74697234
74697056 LOTUSS SANDWICH SPREAD WITH TUNA 185G 4.99MYR https://www.lotuss.com.my/en/product/lotuss-sandwich-spread-with-tuna-185g-74697056
74706551 LOTUSS MURUKU 200G CAMPURAN 3.99MYR https://www.lotuss.com.my/en/product/lotuss-muruku-200g-campuran-74706551
3341828 MUNCHYS OATKRUNCH S/BERRY B/CURRANT 390G 7.99MYR https://www.lotuss.com.my/en/product/munchys-oatkrunch-s-berry-b-currant-390g-3341828
169080 KNIFE COOKING OIL 5KG 44.95MYR https://www.lotuss.com.my/en/product/knife-cooking-oil-5kg-169080
125946 NESTLE HONEY STARS ECONOPACK 500G 14.99MYR https://www.lotuss.com.my/en/product/nestle-honey-stars-econopack-500g-125946