首页
学习
活动
专区
圈层
工具
发布
社区首页 >问答首页 >抓取包含请求的页面不会返回所有html标记

问抓取包含请求的页面不会返回所有html标记
EN

Stack Overflow用户
提问于 2020-05-26 22:58:40
回答 2查看 530关注 0票数 0

为了提取ol id=“page”中的每个li标记的详细信息,我试图抓取这个价格。问题是返回的.html代码有一些空白标记。具体地说,在每个li中,不返回标记div class="shop cf“的内容。为此,我使用了requests和BeautifulSoup,如下所示:

代码语言:javascript
复制
import requests
import time
from bs4 import BeautifulSoup

headers = {
    "Connection": "keep-alive",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36"
}

url = "https://www.skroutz.gr/s/11706397/Guy-Laroche-Linda-Red.html"
page = requests.get(url, headers=headers )

# i also tried the following two commands in order to wait for the page to load
#seconds = 10
#page = requests.get(url, time.sleep(seconds), headers=headers)

soup = BeautifulSoup(page.content, 'html.parser')

eshops_grid = soup.find("ol", id="prices")
eshops_product = eshops_grid.findAll("li", class_='cf card js-product-card')
for eshop in eshops_product[0:]:
    eshop_name = eshop.find("div", class_="shop-name").text
    print(eshop_name) # I need to print the eshop_name for each eshop

虽然我需要使用requests库来完成此任务,但出于这个原因,我也使用了selenium,但同样的问题也会发生。

代码语言:javascript
复制
from selenium import webdriver
from pyvirtualdisplay import Display
from bs4 import BeautifulSoup

# We are opening a browser but not visible
print('- Open a browser but not visible ')
display = Display(visible=0, size=(1920, 1080))
display.start()

driver = webdriver.Chrome("/usr/lib/chromium-browser/chromedriver")

url = 'https://www.skroutz.gr/s/11706397/Guy-Laroche-Linda-Red.html?o=%CE%9C%CF%80%CE%BF%CF%85%CF%81%CE%BD%CE%BF%CF%8D%CE%B6%CE%B9%20Guy%20Laroche%20Linda%20Red'
#print('- Get the initial url of brandwatch')
driver.get(url)

page = driver.page_source
soup = BeautifulSoup(page, 'html.parser')

eshops_grid = soup.find("ol", id="prices")
eshops_product = eshops_grid.findAll("li", class_='cf card js-product-card')
for eshop in eshops_product[0:]:
    eshop_name = eshop.find("div", class_="shop-name").text
    print(eshop_name) # I need to print the eshop_name for each eshop

有没有办法获取每个"li“的所有内容,以便提取并打印”eshop_name“?

EN

回答 2

Stack Overflow用户

发布于 2020-05-31 15:24:56

这是你想要的。

代码语言:javascript
复制
import requests
import demjson
from bs4 import BeautifulSoup as bs

headers = {
    'authority': 'www.skroutz.gr',
    'cache-control': 'max-age=0',
    'sec-ch-ua': '"Google Chrome"; v="83"',
    'sec-ch-ua-mobile': '?0',
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36',
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'sec-fetch-site': 'none',
    'sec-fetch-mode': 'navigate',
    'sec-fetch-user': '?1',
    'sec-fetch-dest': 'document',
    'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
    'cookie': '_helmet_couch=eyJzZXNzaW9uX2lkIjoiNjgzNzhmMmNmNjI5OTcxNjI5NzU2ZWNmMTM5MzE5MmIiLCJidWNrZXRfaWQiOiJmNTk1ZGRhYy00ZmVhLTQ5NmYtODNkNS00OWQzODgzMWFhYTAiLCJsYXN0X3NlZW4iOjE1OTEyNjgwNTUsInZvbCI6MSwiX2NzcmZfdG9rZW4iOiI1a3Yxb3FKTmhXTCs1YUxzdjYzRFk3TlNXeGs5TlhXYmZhM0UzSmtEL0NBPSJ9--22dfbfe582c0f3a7485e20d9d3932b32fbfb721b',
    'if-none-match': 'W/"e6fb8187391e99a90270c2351f9d17cd"',
}

params = (
    ('o', '\u039C\u03C0\u03BF\u03C5\u03C1\u03BD\u03BF\u03CD\u03B6\u03B9 Guy Laroche Linda Red'),
)

response = requests.get('https://www.skroutz.gr/s/11706397/Guy-Laroche-Linda-Red.html', headers=headers, params=params)

data = bs(response.text,'lxml')
s = data.find_all('script')[5].text.split('SKR.page.first_shop_name = ')[1].split(';')[0].replace('"','')
print(s)

输出为:

代码语言:javascript
复制
Spitishop
票数 0
EN

Stack Overflow用户

发布于 2020-06-16 16:26:37

我认为建议的解决方案只适用于第一项。商店列表由通过JS发送的单独的POST请求提供,然后由浏览器单独呈现。

您可以使用适当的头部和POST数据(即在原始请求的"js- product -link content-placeholder“项中找到的产品ID)将POST请求复制到https://www.skroutz.gr/personalization/product_prices.json,返回值为JSON格式。

编辑

好吧,这些信息被“隐藏”在“原始”页面中用base64编码的超文本标记语言注释中。下面的代码用于识别关于商店所需的信息。注释掉的行可以用来发出后续的POST请求,以防需要任何进一步的信息,而这些信息不在base64编码的字符串中(老实说,我没有费心去检查它)。

代码语言:javascript
复制
import requests
from lxml import html
import simplejson as json
import copy

sess = requests.Session()
sess.head('https://www.skroutz.gr')

headers = {
    'authority': 'www.skroutz.gr',
    'cache-control': 'max-age=0',
    'sec-ch-ua': '"Google Chrome"; v="83"',
    'sec-ch-ua-mobile': '?0',
    'upgrade-insecure-requests': '1',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.97 Safari/537.36',
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
    'sec-fetch-site': 'none',
    'sec-fetch-mode': 'navigate',
    'sec-fetch-user': '?1',
    'sec-fetch-dest': 'document',
    'accept-language': 'en-GB,en-US;q=0.9,en;q=0.8',
    'if-none-match': 'W/"e6fb8187391e99a90270c2351f9d17cd"'
}

params = (('o', '\u039C\u03C0\u03BF\u03C5\u03C1\u03BD\u03BF\u03CD\u03B6\u03B9 Guy Laroche Linda Red'),)

response = sess.get('https://www.skroutz.gr/s/11706397/Guy-Laroche-Linda-Red.html', headers=headers, params=params)
tree = html.fromstring(response.content)
# hrefs = tree.xpath('//a[@class="js-product-link content-placeholder"]/@href')
# ids = [x.split('/')[-1] for x in hrefs]
# headers2 = copy.deepcopy(headers)
# headers2['content-type'] = 'application/json'
# ret = sess.post('https://www.skroutz.gr/personalization/product_prices.json', data=json.dumps({'product_ids': ids}), headers=headers2)
details = tree.xpath('//script[@type="application/json"]')[0]
# details.text_content() contains the base64 encoded elements within HTML comments
details_b64 = details.text_content()[4:-3] # strip off the html comments
details_d = json.loads(base64.b64decode(details_b64).decode('utf-8')) # based64 decode the string, decode it in utf-8 and parse the resulting json  object
票数 0
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接:

https://stackoverflow.com/questions/62025117

复制
相关文章

相似问题

领券
问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档