Python bs4 .find没有检测到文章

Question

我试图得到产品的名称，但是当它到达赞助的产品时，它会返回None。这是我的密码

    next_page_url = 'https://www.jumia.com.ng/catalog/?q=oraimo&shipped_from=country_local&page=1#catalog-listing'
    result_nextpage = requests.get(next_page_url, headers=headers).text # headers are generated from default python 'fake_headers' module.
    doc_nextpage = BeautifulSoup(result_nextpage, 'lxml') # I also tried other parsers
    divs = doc_nextpage.find('div', class_='-paxs row _no-g _4cl-3cm-shs')
    result_articles = divs.select('h3.name')
    for i in result_articles:
        print(i.string)

结果；

Oraimo FreePods-3 2Baba Edition BT 5.2 Wireless Stereo Earbuds
Oraimo 27000mAh Massive Power Charing Bank Traveller 3 Byte
Oraimo OPB-P116DN 10000 Mah Power-Bank Dual Fast Charging
Oraimo FreePods3 True Wireless Stereo Earbuds IPX5 & Sweat Proof
Oraimo Smart Watch 1.69'' IPS Screen IP68 Waterproof
Oraimo FreePods-2 2Baba-version True Wireless Earbuds
Oraimo Silver Edition Smart Watch 1.69'' IPS Screen IP68 Waterproof
Oraimo Charger UKDualUSB OCW-U63D White
Oraimo Portable Wireless Speaker Subwoofer Outdoor Sound Box
Oraimo Charger Oraimo UKDualUSB OCW-U81F White
Oraimo Power Oraimo Bank OPB-P206DN 20KmAh
Oraimo SoundPro Wireless Speaker Muti-Model Music Play
Oraimo Tempo-W3 Smart Watch Health Monitor IP67 Waterproof
Oraimo Car Charger Oraimo OCC-21DML Black
Oraimo SoundPro-2C 10W Portable Wireless Bluetooth Speaker
Oraimo Necklace 5C Neckband Wireless Earphone
Oraimo COMPACT 10000mAh Ultra Slim Fast Charging Power Bank
Oraimo 10000mAh OPTIMIZED SLIM Power-bank With LED Light
Oraimo Mermaid Half In-ear Earphone With Mic
Oraimo Necklace 3 Lite Neckband BT 5.0 Wireless Earphone
Oraimo Senior BT5.0 Single Wireless Bluetooth Headsets
Oraimo True Wireless Bluetooth Earbuds- Freepods 2
Oraimo FreePods-2 2Baba-version True Wireless Earbuds
Oraimo Air-Buds-2S Super Bass Wireless Stereo Earbuds
Oraimo 20000MAH Powerbank -long Lasting PowerBank
Oraimo Bluetooth Wireless SOUNDBAR SPEAKER
Oraimo Shark-2 BT5.0 In-Ear Wireless Bluetooth Headphones
Oraimo BoomPop Over-Ear Bluetooth Wireless Headphone
Oraimo  20000MAH Powerbank -long Lasting Power For Days
Oraimo FreePods-2 2Baba-Version  True Wireless Stereo Earbud
Oraimo 2021 Latest Edition Smart Function Waterproof Smart Watch
Oraimo OCW-U36S Efficient And Durable USB Charger - Black
Oraimo FreePods-2 2Baba-Version  True Wireless Stereo Earbud-white
Oraimo 10000MAh Ultimate Slim Power Bank - Black
Oraimo 20000MAH Powerbank - Power For Days
Oraimo 10000mAh Ultra Slim Fast Charging Power Bank
Oraimo 2020 Edition Tempo S - OSW-11 Multi Function Smart Watch
Oraimo SOLID 27000mAh Massive Powerbank OPB-P271D Traveller 3 Byte
Oraimo FreePods-3 2Baba Edition BT 5.2 Wireless Stereo Earbuds
Oraimo Tempo-S IP67 Waterproof Smart Watch WITH AMAZING FUNCTIONS
None
None
None
None
None
None
None
None

文章标签41-48是赞助的产品，产品的名称显示在浏览器中的检查元素中，但是bs4没有检测到它，但是它检测到了其他的非应答。请帮帮忙。

Answer 1

Note首先，看看你的汤/doc_nextpage --这是你处理数据的事实。

会发生什么？

在您的doc_nextpage中，您的赞助产品的html是空的，这就是为什么您得到这些None。

它们是空的，因为它们将由网站动态提供，而请求无法处理这一点。解释/操纵数据的不是浏览器。

怎么修？

一种选择是使用硒模拟浏览器行为，并让page_source用美汤或硒本身处理它。

示例(硒4)

from bs4 import BeautifulSoup 
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service as ChromeService
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

options = webdriver.ChromeOptions()
service = ChromeService(executable_path='ENTER YOUR PATH TO CHROMEDRIVER')
driver = webdriver.Chrome(service=service, options=options)
driver.get('https://www.jumia.com.ng/catalog/?q=oraimo&shipped_from=country_local&page=1#catalog-listing')

WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, '[data-list="sponsored"]')))

soup = BeautifulSoup(driver.page_source, 'lxml')

print([x.text for x in soup.select('article h3.name')])

driver.close()

输出

['Oraimo FreePods-3 2Baba Edition BT 5.2 Wireless Stereo Earbuds',
 'Oraimo 27000mAh Massive Power Charing Bank Traveller 3 Byte',
 'Oraimo OPB-P116DN 10000 Mah Power-Bank Dual Fast Charging',
 'Oraimo FreePods3 True Wireless Stereo Earbuds IPX5 & Sweat Proof',
 "Oraimo Smart Watch 1.69'' IPS Screen IP68 Waterproof",
 'Oraimo FreePods-2 2Baba-version True Wireless Earbuds',
 "Oraimo Silver Edition Smart Watch 1.69'' IPS Screen IP68 Waterproof",
 'Oraimo Charger UKDualUSB OCW-U63D White',
 'Oraimo Portable Wireless Speaker Subwoofer Outdoor Sound Box',
 'Oraimo Charger Oraimo UKDualUSB OCW-U81F White',
 'Oraimo Portable Source 10000mAh Po Wer Ba Nk Oraimo OPB-P110D',
 'Oraimo Power Oraimo Bank OPB-P206DN 20KmAh',
 'Oraimo SoundPro Wireless Speaker Muti-Model Music Play',
 'Oraimo Tempo-W3 Smart Watch Health Monitor IP67 Waterproof',
 'Oraimo Car Charger Oraimo OCC-21DML Black',
 'Oraimo SoundPro-2C 10W Portable Wireless Bluetooth Speaker',
 'Oraimo Necklace 5C Neckband Wireless Earphone',
 'Oraimo 10000mAh OPTIMIZED SLIM Power-bank With LED Light',
 'Oraimo COMPACT 10000mAh Ultra Slim Power Fast Charging Bank',
 'Oraimo Mermaid Half In-ear Earphone With Mic',
 'Oraimo Necklace 3 Lite Neckband BT 5.0 Wireless Earphone',
 'Oraimo Senior BT5.0 Single Wireless Bluetooth Headsets',
 'Oraimo True Wireless Bluetooth Earbuds- Freepods 2',
 'Oraimo Pilot 20000mAh 2.1A Fast  Power Charging Bank',
 'Oraimo FreePods-2 2Baba-version True Wireless Earbuds',
 'Oraimo Air-Buds-2S Super Bass Wireless Stereo Earbuds',
 'Oraimo 20000MAH Powerbank -long Lasting PowerBank',
 'Oraimo Bluetooth Wireless SOUNDBAR SPEAKER',
 'Oraimo Shark-2 BT5.0 In-Ear Wireless Bluetooth Headphones',
 'Oraimo BoomPop Over-Ear Bluetooth Wireless Headphone',
 'Oraimo  20000MAH Powerbank -long Lasting Power For Days',
 'Oraimo FreePods-2 2Baba-Version  True Wireless Stereo Earbud',
 'Oraimo OCW-U36S Efficient And Durable USB Charger - Black',
 'Oraimo 2021 Latest Edition Smart Function Waterproof Smart Watch',
 'Oraimo OCW-U36S Efficient And Durable USB Charger - Black',
 'Oraimo Firefly-2 5.0V/2.1A Dual USB Fast Wall Charger',
 'Oraimo FreePods-2 2Baba-Version  True Wireless Stereo Earbud-white',
 'Oraimo 10000MAh Ultimate Slim Power Bank - Black',
 'Oraimo SOLID 27000mAh Massive Powerbank OPB-P271D Traveller 3 Byte',
 'Oraimo FreePods-3 2Baba Edition BT 5.2 Wireless Stereo Earbuds',
 'Oraimo Massive 27000mAh Travellers 3 Byte OPB-P271D Power Bank',
 'Oraimo 1.69" IPS Screen IP68 Waterproof Smart Watch Pro-Silver',
 'Oraimo Tempo-S IP67 Waterproof Smart Watch WITH AMAZING FUNCTIONS',
 'Oraimo FreePods-3 E104D 2Baba Edition BT 5.2 Wireless Earbuds',
 'Oraimo Tempo-S IP67 Waterproof Smart Watch',
 'Oraimo 2020 Edition Tempo S - OSW-11 Multi Function Smart Watch',
 'Oraimo 10000mAh Ultra Slim Fast Charging Power Bank',
 'Oraimo 20000MAH Powerbank - Power For Days']