Python web scraper打印'None'，我如何修复它？

Question

我正在尝试为学校做一个项目，我取一个给定的股票代码名称，并找到在SeekingAlpha上“观看”它的人数，但当我尝试打印时，我总是得到一个“无”的值。我该如何解决这个问题呢？

这是我第一次尝试网页抓取，但我在BeautifulSoup上做了一些研究，认为这是最好的选择。我也在使用Anaconda环境。在我的代码中，我试图找到股票代码的完整公司名称，以及在SeekingAlpha上关注它的人数。由于某种原因，我能够检索到股票行情的公司名称，但当我尝试打印关注者的数量时，它显示为“无”。我尝试了我能想到的所有变体来寻找追随者，但它们都没有结果。

Here is the HTML: (Here I want the value 83,530)

下面是我的代码：

import requests
import urllib.request as urllib2
from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
from lxml import etree

listOfTickers = ["ATVI", "GOOG", "AAPL", "AMZN", "BRK.B", "BRK.A", "NFLX", "SNAP"]

for i in range(len(listOfTickers)):
    ticker = listOfTickers[i]
    quotePage = Request("https://seekingalpha.com/symbol/" + ticker, headers = {'User-Agent': 'Mozilla/5.0'})

    page = urlopen(quotePage).read()

    soup = BeautifulSoup(page, "lxml")

    company_name = soup.find("div", {"class" :"ticker-title"})
    followers_number = soup.find('div', {"class":"followers-number"})

    company= company_name.text.strip()
    #followers = followers_number.text.strip()

    print(followers_number)
    print(company)

​

这是我的结果

​

​

Answer 1

尝试执行以下操作以获得所需的输出。您希望抓取的内容是动态生成的，因此request module或urllib不会有任何帮助。您可以使用任何浏览器模拟器或requests-html来完成此任务。也没有必要使用BeautifulSoup。但是，我将其保留为原样只是因为您一开始就使用了它。

from requests_html import HTMLSession
from bs4 import BeautifulSoup

tickers = ["ATVI", "GOOG", "AAPL", "AMZN"]

with HTMLSession() as session:
    for i in range(len(tickers)):
        quotePage = session.get("https://seekingalpha.com/symbol/{}".format(tickers[i]))
        quotePage.html.render(5)
        soup = BeautifulSoup(quotePage.html.html, "lxml")
        followers_number = soup.find(class_="followers-number")
        print(followers_number)

输出结果可能如下所示：

<div class="followers-number">(<span>83,532</span> followers)</div>
<div class="followers-number" title="1,032,510">(<span>1.03M</span> followers)</div>
<div class="followers-number" title="2,065,199">(<span>2.07M</span> followers)</div>

Answer 2

只需使用与页面相同的端点来检索订阅信息，包括# followers：

import requests

tickers = [ "atvi", "goog", "aapl", "amzn", "brk.b", "brk.a", "nflx", "snap"]

with requests.Session() as s:
    for ticker in tickers:
        r = s.get('https://seekingalpha.com/memcached2/get_subscribe_data/{}?id={}'.format(ticker, ticker)).json()
        print(ticker, r['portfolio_count'])

​

​

Answer 3

由于关注者是通过ajax加载的，所以BeautifulSoup无法访问它的值。使用像selineum/Phantojs这样的无头浏览器，你可以得到完整的html，包括javascript生成的内容。另一种方式是通过向端点发出额外的请求，在这些端点中，javascript会渲染页面的某些部分。下面是一个可行的解决方案

import requests
import urllib.request as urllib2
from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
from lxml import etree

listOfTickers = ["ATVI", "GOOG", "AAPL", "AMZN", "BRK.B", "BRK.A", "NFLX", "SNAP"]

def getFollowersCount(ticker):

    # build url 
    url = 'https://seekingalpha.com/memcached2/get_subscribe_data/{}?id={}'.format(ticker.lower(), ticker.lower())

    # Using requests module not urllib.request.
    counter = requests.get(url)

    # If requests return is json than return portfolio_count otherwise 0
    try:
        return counter.json()['portfolio_count']
    except:
        return 0

for ticker in listOfTickers:

    quotePage = Request("https://seekingalpha.com/symbol/" + ticker, headers = {'User-Agent': 'Mozilla/5.0'})

    page = urlopen(quotePage).read()

    soup = BeautifulSoup(page, "lxml")

    company_name = soup.find("div", {"class" :"ticker-title"})
    #followers_number = soup.find('div', {"class":"followers-number"})
    followers_number = getFollowersCount(ticker)

    company= company_name.text.strip()
    #followers = followers_number.text.strip()

    print(followers_number)
    print(company)