用python实现facebook的网络爬虫

Question

我正在尝试使用python中的web-Crawler来打印facebook推荐器的数量。例如，在这篇来自天空新闻(http://news.sky.com/story/1330046/are-putins-little-green-men-back-in-ukraine)的文章中，有大约60个facebook推荐。我想用网络爬虫在python程序中打印这个数字。我试着这样做，但它没有打印任何内容：

import requests
from bs4 import BeautifulSoup

def get_single_item_data(item_url):
    source_code = requests.get(item_url)
    plain_text = source_code.text
    soup = BeautifulSoup(plain_text)
    # if you want to gather information from that page
    for item_name in soup.findAll('span', {'class': 'pluginCountTextDisconnected'}):
        try:
                print(item_name.string)
        except:
                print("error")

get_single_item_data("http://news.sky.com/story/1330046/are-putins-little-green-men-back-in-ukraine")

Answer 1

Facebook建议在iframe.中加载，您可以按照iframe src属性加载到该页面，然后加载span.pluginCountTextDisconnected的文本：

import requests
from bs4 import BeautifulSoup

url = 'http://news.sky.com/story/1330046/are-putins-little-green-men-back-in-ukraine'
r = requests.get(url) # get the page through requests
soup = BeautifulSoup(r.text) # create a BeautifulSoup object from the page's HTML

url = soup('iframe')[0]['src'] # search for the iframe element and get its src attribute
r = requests.get('http://' + url[2:]) # get the next page from requests with the iframe URL
soup = BeautifulSoup(r.text) # create another BeautifulSoup object

print(soup.find('span', class_='pluginCountTextDisconnected').string) # get the directed information

由于src属性返回//www.facebook.com/plugins/like.php?href=http%3A%2F%2Fnews.sky.com%2Fstory%2F1330046&send=false&layout=button_count&width=120&show_faces=false&action=recommend&colorscheme=light&font=arial&height=21，因此第二个requests.get也是这样编写的。我添加了http://，忽略了前导的//。

BeautifulSoup documentation

Requests documentation

Answer 2

Facebook的推荐是从javascript动态加载的，所以你的HTML解析器不能使用它们。您需要使用Graph API和FQL直接从Facebook获取答案。

生成访问令牌后可以在其中浏览查询的Here is a web console。