BeautifulSoup传递div

Question

我想传递这一页：

http://animedigitalnetwork.fr/video/naruto-shippuden

我测试这个：

import urllib2
from BeautifulSoup import BeautifulSoup

page = urllib2.urlopen('http://animedigitalnetwork.fr/video/naruto-shippuden')
soup = BeautifulSoup(page)
first_div = soup.find('div',{"class" : "adn-video"})

结果不是鸣人!？

<div class="adn-video"> <div class="adn-video_screenshot">
<img src="http://image.animedigitalnetwork.fr/license/claymore/tv/web/eps1_328x184.jpg" alt="Claymore 1" /><span class="adn_video_play-button"></span> </div><div class="adn-video_text"><div class="adn-video_title">
<h4>Claymore</h4><span>Épisode 1</span><div class="adn-rating mobile-hide" itemprop="aggregateRating" itemscope="itemscope" itemtype="http://schema.org/AggregateRating"><meta itemprop="ratingValue" content="4.6667" /><meta itemprop="ratingCount" content="10" /><div id="adn-rating"><ul class="adn-rating_empty"><li>&#xe002;</li><li>&#xe002;</li><li>&#xe002;</li><li>&#xe002;</li><li>&#xe002;</li></ul><ul class="adn-rating_rating"><li>&#xe002;</li><li>&#xe002;</li><li>&#xe002;</li><li>&#xe002;</li><li>&#xe002;</li></ul></div><p class="adn-rating-message"></p></div></div><div class="adn-video_link">
<a title="Claymore 1" href="/video/claymore/1849-episode-1-la-claymore">Voir la vidéo</a>
</div></div></div>

Answer 1

这里有多个问题：

• 在形成视频列表时还有一个额外的异步调用：
  • 首先，您需要从初始页面中提取playlist参数值。
  • 然后，发出一个帖子请求将视频发送到这个url

​

• 您正在使用过时和未维护的版本 of BeautifulSoup -切换到bs4： pip安装beautifulsoup4
• 使用requests而不是urllib2，维护会话并传递请求头

完整的工作守则：

from bs4 import BeautifulSoup
import requests

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'}

# initialize session
session = requests.Session()

# getting playlist
response = session.get('http://animedigitalnetwork.fr/video/naruto-shippuden', headers=headers)
soup = BeautifulSoup(response.content)
playlist = soup.find('a', {'data-playlist': True})['data-playlist']

# getting list of videos
url = 'http://animedigitalnetwork.fr/index.php?option=com_vodvideo&view=playlist&format=raw'
response = session.post(url, data={
    'playlist': playlist,
    'season': '',
    'order': 'DESC'
}, headers=headers)

soup = BeautifulSoup(response.content)
for video in soup.select('div.adn-video'):
    print video.a.get('title')

印刷品(视频标题清单)：

Naruto Shippuden 391
Naruto Shippuden 390
Naruto Shippuden 389
...
Naruto Shippuden 3
Naruto Shippuden 2
Naruto Shippuden 1