如何使用python和漂亮的汤来抓取标题和描述？

Question

问题概述：

链接：https://www.bobfinancial.com/eterna.jsp

在细节部分:基本上我想要所有的要点。

详情：

[ #This is an array of Strings... 
    
"Milestone Rewards: Earn 10,000 bonus reward points on spending ₹ 50,000 within 60 days & 20,000 bonus reward points on spending ₹ 5,00,000 in a year.",
"Fuel Surcharge Waiver*: 1% fuel surcharge waiver at all fuel stations across India on transactions between Rs.400 and Rs.5,000 (Max. Rs. 250 per statement cycle). Note -No Reward Points are earned on fuel transactions.",
"Core Reward Points:  3 reward points for every ₹ 100 spent on any other category.",
"Redeem reward points for cashback: Redeem your reward points as cashback and other exciting options.All your accumulated reward points can be redeemed for cashback @ 1 reward point = ₹ 0.25",
"In-built insurance cover: Get free Personal Accidental Death Cover to ensure financial protection of your family (Air: 1 Crs, Non-Air: 10 Lakhs) ",
"Zero liability on lost card. Report loss of card immediately to ensure zero liability on any fraudulent transactions",
"Easy EMI option. Convert purchase of > 2,500/- on your card into easy EMIs of 6/12 months"
]

实际问题:

链接到刮除：https://www.online.citibank.co.in/credit-card/rewards/citi-rewards-credit-card?eOfferCode=INCCCCTWAFCTRELM

我希望从该链接中获得的项目在以下图像中：

​

​

from urllib.request import urlopen
from bs4 import BeautifulSoup
import json,requests

details = []

url = ['https://www.online.citibank.co.in/credit-card/rewards/citi-rewards-credit-card?eOfferCode=INCCCCTWAFCTRELM']
html = urlopen(url[0])
soup = BeautifulSoup(html, 'lxml')
addr= soup.find_all('span',class_ = 'm-bottom-0 header-4 font-weight-bold display-text')
print(addr)

我编写了上述代码，并在得到以下输出后陷入困境：

[]

我不知道如何继续和刮我想要的信息，任何帮助都是非常感谢的。

Answer 1

您可以只搜索具有display-text类的标题。然后主体就是下面的<span>条目。这避免了任何硬编码偏移，这可能会破坏其他类似的网页。例如：

from bs4 import BeautifulSoup
import requests
    
url = ['https://www.online.citibank.co.in/credit-card/rewards/citi-rewards-credit-card?eOfferCode=INCCCCTWAFCTRELM']
html = requests.get(url[0])
soup = BeautifulSoup(html.content, 'lxml')
data = []

for span in soup.select('span.m-bottom-0.display-text.font-weight-bold'):
    data.append([span.get_text(strip=True), span.find_next('span').get_text(strip=True)])

print(data)

这将为您提供一个数据结构，用于保存标题和如下描述：

[
    ['Citi Rewards Credit Card', 'Make your shopping more rewarding'], 
    ['Make your shopping more rewarding', 'Get up to 2500 welcome reward points*'], 
    ['Accelerated rewards', 'Earn minimum 1 reward point for every 125 on all purchases.\nEarn 10X reward points at online and physical department and apparel stores.'], 
    ['Bonus rewards', 'Get 300 bonus points on card purchase of INR 30,000 or more in a month'], 
    ['Evergreen reward points', "Redeem now or keep collecting – it's a choice you have because your reward points will never expire."], 
    ['Tap n Pay', 'Now pay the easy way by enabling contactless payments on your Citi credit card.Click hereto see how.'], 
    ['Rewards', "Redeem rewards: Redeem now or keep collecting – it's a choice you have.Click hereto see how"], 
    ['Travel and Lifestyle Services', 'Contact the Travel and Lifestyle specialist to create and plan an experience that will help you enjoy the best in life. Simply call 0008 004 407 027 for Mastercard®cardholders or 1800-114-999 for Visa cardholders.']
]

您可以跳过使用print(data[2:])的前两个结果(如果不需要的话)

它可以扩展到多个URL，头和描述可以组合在一起：

from bs4 import BeautifulSoup
import requests
    
urls = [
    'https://www.online.citibank.co.in/credit-card/fuel-card/citi-indianoil-card?eOfferCode=INCCCCTWAFCTIOPLM',
    'https://www.online.citibank.co.in/credit-card/rewards/citi-rewards-credit-card?eOfferCode=INCCCCTWAFCTRELM',
]

for url in urls:
    html = requests.get(url)
    soup = BeautifulSoup(html.content, 'lxml')
    data = []

    for span in soup.select('span.m-bottom-0.display-text.font-weight-bold'):
        data.append(f'{span.get_text(strip=True)}: {span.find_next("span").get_text(strip=True)}')

    # Display the list of strings
    print('\n'.join(data[1:]))
    print()

为两个URL提供以下输出：

Accelerated Rewards on Fuel spends: 4 Turbo points / Rs. 150 spent &1% fuel surcharge reversalon fuel purchase atauthorized IndianOil outlets^
Earn on all Daily Spends: 2 Turbo points / Rs. 150 spent on groceries and supermarkets#1 Turbo point / Rs. 150 on all other spends.#Clickherefor the details
Redeem Instantly: 1 Turbo Point = Re. 1 of free fuelRedeem your Turbo points instantly via SMS for freeFuel atauthorized IndianOil outlets^
Tap n Pay: Now pay the easy way by enabling contactless payments on your Citi credit card.Click hereto see how.
Redeem Rewards!: Choose from a range of redeeming options including fuel, holidays, air miles, cash back and more.Click hereto see how
Dining Privileges: Up to 20% savings across participating restaurants. To find a Citibank partner restaurant near you,click here
Travel and Lifestyle Services: Contact the Travel and Lifestyle specialist to create and plan an experience that will help you enjoy the best in life. Simply call 0008 004 407 027 for Mastercard cardholders or 1800-114-999 for Visa cardholders.

Accelerated rewards: Earn minimum 1 reward point for every 125 on all purchases.
Earn 10X reward points at online and physical department and apparel stores.
Bonus rewards: Get 300 bonus points on card purchase of INR 30,000 or more in a month
Evergreen reward points: Redeem now or keep collecting – it's a choice you have because your reward points will never expire.
Tap n Pay: Now pay the easy way by enabling contactless payments on your Citi credit card.Click hereto see how.
Rewards: Redeem rewards: Redeem now or keep collecting – it's a choice you have.Click hereto see how
Travel and Lifestyle Services: Contact the Travel and Lifestyle specialist to create and plan an experience that will help you enjoy the best in life. Simply call 0008 004 407 027 for Mastercard®cardholders or 1800-114-999 for Visa cardholders.

Answer 2

试试这个，希望它能帮上忙

from urllib.request import urlopen
from bs4 import BeautifulSoup
import json,requests

details = []

url = ['https://www.online.citibank.co.in/credit-card/rewards/citi-rewards-credit-card?eOfferCode=INCCCCTWAFCTRELM']
html = requests.get(url[0])
print(html.status_code)
soup = BeautifulSoup(html.content, 'lxml')
x = soup.select('span.m-bottom-0')

addr= soup.select('span.m-bottom-0')[12:20] # number of span
for d in addr:
    print(d.get_text())

addr= soup.select('span.m-bottom-0')[58:70]
for d in addr:
    print(d.get_text()) # get_text() method for inner tag text