在使用BeautifulSoup抓取shopee.co.id时获得了一个空列表

Question

我试图用BeautifulSoup包抓取https://shopee.co.id/top_products上排名靠前的产品的名称和销售量。但是无论我怎么尝试，它总是返回一个空的产品列表。你能解释一下在引擎盖下发生了什么吗？下面是我的代码：

import requests
from bs4 import BeautifulSoup as bs

headers = {
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36',
    'cookie': '_gcl_au=1.1.961206468.1594951946; _med=refer; _fbp=fb.2.1594951949275.1940955365; SPC_IA=-1; SPC_F=y1evilme0ImdfEmNWEc08bul3d8toc33; REC_T_ID=fab983c8-c7d2-11ea-a977-ccbbfe23657a; SPC_SI=uv1y64sfvhx3w6dir503ixw89ve2ixt4; _gid=GA1.3.413262278.1594951963; SPC_U=286107140; SPC_EC=GwoQmu7TiknULYXKODlEi5vEgjawyqNcpIWQjoxjQEW2yJ3H/jsB1Pw9iCgGRGYFfAkT/Ej00ruDcf7DHjg4eNGWbCG+0uXcKb7bqLDcn+A2hEl1XMtj1FCCIES7k17xoVdYW1tGg0qaXnSz0/Uf3iaEIIk7Q9rqsnT+COWVg8Y=; csrftoken=5MdKKnZH5boQXpaAza1kOVLRFBjx1eij; welcomePkgShown=true; _ga=GA1.1.1693450966.1594951955; _dc_gtm_UA-61904553-8=1; REC_MD_30_2002454304=1595153616; _ga_SW6D8G0HXK=GS1.1.1595152099.14.1.1595153019.0; REC_MD_41_1000044=1595153318_0_50_0_49; SPC_R_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="; SPC_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_R_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="'
}

shopee_url = 'https://shopee.co.id/top_products'
response = requests.get(shopee_url, headers=headers)
soup = bs(response.text, "html.parser")

products = soup.find_all("._3S8sOC _2QfAXF")
print(type(products))
print(products)

感谢你读到这里。我读过一些类似的答案，但它们似乎不适用于这里。

Answer 1

如果您愿意使用selenium，您可以使用selenium进行请求：

from selenium import webdriver
from bs4 import BeautifulSoup as bs

headers = {
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36',
    'cookie': '_gcl_au=1.1.961206468.1594951946; _med=refer; _fbp=fb.2.1594951949275.1940955365; SPC_IA=-1; SPC_F=y1evilme0ImdfEmNWEc08bul3d8toc33; REC_T_ID=fab983c8-c7d2-11ea-a977-ccbbfe23657a; SPC_SI=uv1y64sfvhx3w6dir503ixw89ve2ixt4; _gid=GA1.3.413262278.1594951963; SPC_U=286107140; SPC_EC=GwoQmu7TiknULYXKODlEi5vEgjawyqNcpIWQjoxjQEW2yJ3H/jsB1Pw9iCgGRGYFfAkT/Ej00ruDcf7DHjg4eNGWbCG+0uXcKb7bqLDcn+A2hEl1XMtj1FCCIES7k17xoVdYW1tGg0qaXnSz0/Uf3iaEIIk7Q9rqsnT+COWVg8Y=; csrftoken=5MdKKnZH5boQXpaAza1kOVLRFBjx1eij; welcomePkgShown=true; _ga=GA1.1.1693450966.1594951955; _dc_gtm_UA-61904553-8=1; REC_MD_30_2002454304=1595153616; _ga_SW6D8G0HXK=GS1.1.1595152099.14.1.1595153019.0; REC_MD_41_1000044=1595153318_0_50_0_49; SPC_R_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="; SPC_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_R_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="'
}
driver = webdriver.Firefox(executable_path='c:/program/geckodriver.exe')
shopee_url = 'https://shopee.co.id/top_products'
driver.get(shopee_url)
driver.implicitly_wait(15)
response = driver.page_source
driver.close()
soup = bs(response, "html.parser")

products = soup.select(".shopee-item-card__lower-padding")

#print(products)

for product in products:
    name = product.select_one('.shopee-item-card__text-name span')
    print(name.get_text(strip=True))
    price = product.select_one('.shopee-item-card__current-price')
    print(price.text)

注您需要安装Firefox、geckodriver和selenium。

输出：

Kuota Telkomsel Cek Dadakan
Rp300
Aigo 1GB 5Hari
Rp7.375
Paket Data Tri Mix Small 5GB
Rp4.500
Paket Data Tri Mix Small 2.75GB
Rp9.500
*PROMO MURAH* SURPRISE DEAL PAKET DATA INTERNET KUOTA TELKOMSEL SIMPATI AS LOOP OMG 20GB 30GB 50GB
Rp1.000
Kuota Telkomsel Cek Dadakan
Rp500
Paket Data Telkomsel 12GB
Rp98.000
Paket Data Tri AON Unlimited + 6GB
Rp58.000
KUOTA INTERNET PAKET DATA INTERNET TELKOMSEL KUOTA DATA FLASH Kartu Simpati As Loop (KUOTA MURAH)
Rp1.000
[MURAH] PAKET DATA TELKOMSEL ∣ KUOTA INTERNET ∣ PAKET DATA ∣ SIMPATI  ∣ AS  ∣ LOOOP
Rp1.000
Paket Data Tri AddOn 12GB
Rp48.000
Paket Data  Indosat Freedom U 7GB+20GB Apps / 30 Hari
Rp66.500
PAKET DATA TELKOMSEL ISI ULANG SIMPATI 14GB/27GB/37GB - AS 6,5GB/10GB/17GB/32GB - LOOP 3GB/8GB/15GB
Rp38.000
Paket Data Indosat Freedom Internet 4GB / 30 Hari
Rp24.000
Paket Data Indosat Freedom U 1GB+4.5GB Apps / 30 Hari
Rp24.000
KUOTA TELKOMSEL paket kuota data internet telkomsel termurah 52gb injek injeck kuota  promo
Rp1.000
Kuota Telkomsel Simpati 52GB 50GB 24jam Langsung Masuk
Rp1.000
Paket Data Telkomsel 50GB
Rp195.000
Paket Data Indosat Freedom Internet 10GB /30 Hari
Rp48.000
KUOTA TELKOMSEL paket data internet telkomsel kuota internet TELKOMSEL  termurah simpati as loop
Rp1.000
Paket Data Telkomsel Data 25.000
Rp23.500
VOCHER VOUCHER KUOTA AXIS AIGO 1GB 5HARI NASIONAL
Rp7.390
PAKET DATA KUOTA INDOSAT [DISKON 15%] TERMURAH
Rp1.000
VOUCHER Axis 1gb 5 hari 24 Jam KUOTA 1 GB
Rp7.500
Paket Data Indosat Freedom Combo 14GB, 30 Hari
Rp48.500
[PROMO] KUOTA BESAR PAKET OMG TELKOMSEL MURAH
Rp1.000
Paket Data Indosat Freedom Combo 8GB/ 30 Hari
Rp33.500
TELKOMSEL PEMBAYARAN RESELLER/ AGEN atau Penambahan Selisih Harga Produk ThalhahCell
Rp1.000
Aigo 3GB 15Hari
Rp18.500
Paket Data Indosat Freedom U 3GB+15GB Apps / 30 Hari
Rp49.900