出现错误的BeautifulSoup响应

Question

我正试着用废话把我的脚弄湿。我试着用自己的方式浏览文档，但在第一步我就遇到了一个问题。

这是我的代码：

from bs4 import BeautifulSoup
soup = BeautifulSoup('https://api.flickr.com/services/rest/?method=flickr.photos.search&api_key=5....1b&per_page=250&accuracy=1&has_geo=1&extras=geo,tags,views,description')

print(soup.prettify())

这是我得到的回应：

Warning (from warnings module):
File "/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/bs4/__init__.py", line 189
'"%s" looks like a URL. Beautiful Soup is not an HTTP client. You should probably use an     
HTTP client to get the document behind the URL, and feed that document to Beautiful Soup.' % markup)
UserWarning: "https://api.flickr.com/services/rest/?method=flickr.photos.search&api_key=5...b&per_page=250&accuracy=1&has_geo=1&extras=geo,tags,views,description" 
looks like a URL. Beautiful Soup is not an HTTP client. You should 
probably use an HTTP client to get the document behind the URL, and feed that document    
to Beautiful Soup.
https://api.flickr.com/services/rest/?method=flickr.photos.search&api;_key=5...b&per;_page=250&accuracy;=1&has;_geo=1&extras;=geo,tags,views,description

是因为我试图调用http**s**，还是另一个问题？谢谢你的帮忙!

Answer 1

您正在将URL作为字符串传递。相反，您需要通过urllib2或requests获取页面源代码

from urllib2 import urlopen  # for Python 3: from urllib.request import urlopen
from bs4 import BeautifulSoup

URL = 'https://api.flickr.com/services/rest/?method=flickr.photos.search&api_key=5....1b&per_page=250&accuracy=1&has_geo=1&extras=geo,tags,views,description'
soup = BeautifulSoup(urlopen(URL))

注意，你不需要对urlopen()的结果调用read()，BeautifulSoup允许第一个参数是一个类似文件的对象，urlopen()返回一个类似文件的对象。

Answer 2

这个错误说明了一切，您正在向Beautiful Soup传递一个URL。您需要首先获取网站内容，然后才将内容传递给BS。

要下载内容，可以使用urlib2

import urllib2
response = urllib2.urlopen('http://www.example.com/')
html = response.read()

以及以后的

soup = BeautifulSoup(html)