Python - BeautifulSoup不会生成相同的代码

Question

我是python的新手，我想在这个网站上收集一些信息：

import requests

from bs4 import BeautifulSoup

URL = 'https://www.leroymerlin.com.br/piso-ceramico-interno-madeira-esmaltado-borda-arredondada-reale-noce-61x61cm-rosagres_89371261'
page = requests.get(URL)
soup = BeautifulSoup(page.content)
mydivs = soup.findAll("div", {"class": "carousel"})

但是mydiv是空的，但是在view-source中有这一行。同样：

for link in soup.find_all('a'):
    print(link.get('href'))

给我那些我在view-source中看不到的链接，也不要给我一些我在view-source中看到的链接。我认为这是一个iframe问题，但我尝试了，但我不能解决它。

有人能帮帮我吗？

Answer 1

我认为这里的问题是python正在接收一个不同的网页，因为它被重定向到https://www.leroymerlin.com.br/localizacao。

为了到达你正在抓取的网页，你应该设置区域cookie，这样你就不会被重定向：

page = requests.get(URL, cookies={'region': 'eyJpdiI6IlN1SUhieDhmUGtxbDVzcHIyZVgrSUE9PSIsInZhbHVlIjoiZHhycHE0MGRlVWNkNHk0U2NDSCtCZz09IiwibWFjIjoiMmFiMjVhZTFjMjFhNWU4NzkyZmFiNTUxOGM5MWYzMmIzYmJiYzNkMDk4NDJjYWFiYTdkNDVjNjMzYzVjNWNhMCJ9'})

我使用eyJpdiI6IlN1SUhieDhmUGtxbDVzcHIyZVgrSUE9PSIsInZhbHVlIjoiZHhycHE0MGRlVWNkNHk0U2NDSCtCZz09IiwibWFjIjoiMmFiMjVhZTFjMjFhNWU4NzkyZmFiNTUxOGM5MWYzMmIzYmJiYzNkMDk4NDJjYWFiYTdkNDVjNjMzYzVjNWNhMCJ9作为区域cookie，您可以查看web浏览器开发控制台中的数据包来检查您使用的cookie

​

下面是对此的回应：

>>> mydivs = soup.findAll("div", {"class": "carousel"})
>>> mydivs
/*
* 提示：该行代码过长，系统自动注释不进行高亮。一键复制会移除系统注释 
* [<div class="carousel" data-carousel-tracking-value="89371261" data-items='[{"alt":"Piso Cer\u00e2mico Interno Madeira Esmaltado Borda Arredondada Reale Noce 61x61cm Rosagr\u00eas","url":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_c1d5_600x600.jpeg","zoomDimensions":"1800","zoomUrl":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_c1d5_1800x1800.jpeg","shouldLoadImageZoom":true,"thumb":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_c1d5_52x52.jpeg","youtubeId":""},{"alt":"Piso Cer\u00e2mico Esmaltado Borda Arredondada 61x61cm modelo Reale Noce Rosagr\u00eas","url":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_esmaltado_brilhante_61x61cm_modelo_reale_noce_rosagres_89371261_ed4d_600x600.jpg","zoomDimensions":"600","zoomUrl":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_esmaltado_brilhante_61x61cm_modelo_reale_noce_rosagres_89371261_ed4d_600x600.jpg","shouldLoadImageZoom":true,"thumb":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_esmaltado_brilhante_61x61cm_modelo_reale_noce_rosagres_89371261_ed4d_52x52.jpg","youtubeId":""},{"alt":"Piso Cer\u00e2mico Interno Madeira Esmaltado Borda Arredondada Reale Noce 61x61cm Rosagr\u00eas","url":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_4550_600x600.jpeg","zoomDimensions":"1800","zoomUrl":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_4550_1800x1800.jpeg","shouldLoadImageZoom":true,"thumb":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_4550_52x52.jpeg","youtubeId":null},{"alt":"Piso Cer\u00e2mico Interno Madeira Esmaltado Borda Arredondada Reale Noce 61x61cm Rosagr\u00eas","url":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_586b_600x600.jpeg","zoomDimensions":"1800","zoomUrl":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_586b_1800x1800.jpeg","shouldLoadImageZoom":true,"thumb":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_586b_52x52.jpeg","youtubeId":null},{"alt":"Piso Cer\u00e2mico Interno Madeira Esmaltado Borda Arredondada Reale Noce 61x61cm Rosagr\u00eas","url":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_51ba_600x600.jpeg","zoomDimensions":"1800","zoomUrl":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_51ba_1800x1800.jpeg","shouldLoadImageZoom":true,"thumb":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_51ba_52x52.jpeg","youtubeId":null},{"alt":"Piso Cer\u00e2mico Interno Madeira Esmaltado Borda Arredondada Reale Noce 61x61cm Rosagr\u00eas","url":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_a684_600x600.jpeg","zoomDimensions":"1800","zoomUrl":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_a684_1800x1800.jpeg","shouldLoadImageZoom":true,"thumb":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_a684_52x52.jpeg","youtubeId":null},{"alt":"Piso Cer\u00e2mico Interno Madeira Esmaltado Borda Arredondada Reale Noce 61x61cm Rosagr\u00eas","url":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_09b7_600x600.jpeg","zoomDimensions":"1800","zoomUrl":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_09b7_1800x1800.jpeg","shouldLoadImageZoom":true,"thumb":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_09b7_52x52.jpeg","youtubeId":null},{"alt":"Piso Cer\u00e2mico Interno Madeira Esmaltado Borda Arredondada Reale Noce 61x61cm Rosagr\u00eas","url":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_8e45_600x600.jpeg","zoomDimensions":"1800","zoomUrl":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_8e45_1800x1800.jpeg","shouldLoadImageZoom":true,"thumb":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_8e45_52x52.jpeg","youtubeId":null},{"alt":"Piso Cer\u00e2mico Brilhante Borda Arredondado Reale Noce 61x61cm Rosagr\u00eas","url":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_borda_arredondado_reale_noce_61x61cm_rosagres_89371261_0001_600x600.jpg","zoomDimensions":"600","zoomUrl":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_borda_arredondado_reale_noce_61x61cm_rosagres_89371261_0001_600x600.jpg","shouldLoadImageZoom":true,"thumb":"https:\/\/cdn.leroymerlin.com.br\/products\/piso_ceramico_borda_arredondado_reale_noce_61x61cm_rosagres_89371261_0001_52x52.jpg","youtubeId":null}]' data-product-images-carousel="" data-sketchfab-id="7a24a14dabc44227b465d49673d144e4" data-sketchfab-placeholder="https://cdn.leroymerlin.com.br/assets/images/sketchfab-placeholder.svg" data-sketchfab-thumb="https://cdn.leroymerlin.com.br/assets/images/sketchfab-3d-model.svg">
*/
<div>
<picture>
<source media="(max-width: 768px)" srcset="https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_c1d5_220x220.jpeg"/>
<source media="(min-width: 769px)" srcset="https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_c1d5_300x300.jpeg"/>
<img alt="Piso Cerâmico Interno Madeira Esmaltado Borda Arredondada Reale Noce 61x61cm Rosagrês" class="image" src="https://cdn.leroymerlin.com.br/products/piso_ceramico_esmaltado_reale_noce_61x61cm_rosagres_89371261_c1d5_300x300.jpeg"/>
</picture>
</div>
</div>]
>>>