使用Newspaper3k时从html中删除嵌入的推文

Question

我正在使用Newspaper3k从在线新闻中提取文本。

from newspaper import Article

urlw = 'https://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=12307959'
article = Article(urlw)
article.download()
article.parse()
string1 = article.text

但是，我可以看到有多条嵌入式tweet是我分析时不需要的。我试着把它们归结为以下几点。

import requests
from bs4 import BeautifulSoup

r = requests.get('https://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=12307959')
soup = BeautifulSoup(r.content, "html.parser")
article_soup = [s.get_text() for s in soup.find_all('p', {'dir': 'ltr'})]

但是，我想不出从string1中删除它们的方法

Answer 1

要使用漂亮的汤删除html标记，只需找到html标记并在html变量上调用extract()。然后，使用soup对象查找文章内容

import requests
from bs4 import BeautifulSoup

r = requests.get('https://www.nzherald.co.nz/nz/news/article.cfm?c_id=1&objectid=12307959')
r.raise_for_status() # check for 4xx + 5xx status code
soup = BeautifulSoup(r.text, "html.parser")

for tweet in soup.find_all('div', {'element-oembed'}):
    tweet.extract() # remove div with class 'element-oembed'

articleTag = soup.find(id='article-content')
print(articleTag.text.strip()) 

输出：

'Traffic is backed up for about 9km after an incident near Spaghetti Junction.  The incident happened about 12pm at the Southern Motorway link to the Northwestern Motorway, westbound.  Drivers were asked to avoid the area and consider using an alternative route.     The New Zealand Transport Agency said at 1.25pm the road had reopened but traffic remained heavy between Penrose and the State Highway 1 link - a journey of about 9km.   Advertisement   Advertise with NZME.     "Consider delaying your journey if possible, or be prepared for delays."  Police have cordoned off a section of footpath on Alex Evans Road, above the motorway, in relation to the incident.'