bs4按id查找表，返回'None‘

Question

不确定为什么这不起作用:(我可以从这个页面中拉出其他表，但不是这个。

import requests
from bs4 import BeautifulSoup as soup

url = requests.get("https://www.basketball-reference.com/teams/BOS/2018.html",
                   headers={'User-Agent': 'Mozilla/5.0'})

page = soup(url.content, 'html')
table = page.find('table', id='team_and_opponent')
print(table)

感谢你的帮助。

Answer 1

页面是动态的。所以在这种情况下你有两个选择。

旁注:如果您看到<table>标记，请不要使用BeautifulSoup，pandas可以通过使用pd.read_html()为您完成这项工作(它实际上在幕后使用bs4

1)首先使用selenium呈现页面，然后使用BeautifulSoup提取<table>标记

2)这些表在html中的注释标签内。你可以用BeautifulSoup提取评论，然后用'table'抓取评论。

我选择了选项2。

import requests
from bs4 import BeautifulSoup
from bs4 import Comment
import pandas as pd


url = 'https://www.basketball-reference.com/teams/BOS/2018.html'
response = requests.get(url)

soup = BeautifulSoup(response.text, 'html.parser')

comments = soup.find_all(string=lambda text: isinstance(text, Comment))

tables = []
for each in comments:
    if 'table' in each:
        try:
            tables.append(pd.read_html(each)[0])
        except:
            continue

我不知道您想要哪个特定的表，但它们就在tables列表中

*输出：**

print (tables[1])
   Unnamed: 0     G     MP     FG    FGA  ...    STL     BLK   TOV     PF    PTS
0        Team  82.0  19805   3141   6975  ...    604     373  1149   1618   8529
1      Team/G   NaN  241.5   38.3   85.1  ...    7.4     4.5  14.0   19.7  104.0
2     Lg Rank   NaN     12     25     25  ...     23      18    15     17     20
3   Year/Year   NaN   0.3%  -0.9%  -0.0%  ...  -2.1%    9.7%  5.6%  -4.0%  -3.7%
4    Opponent  82.0  19805   3066   6973  ...    594     364  1159   1571   8235
5  Opponent/G   NaN  241.5   37.4   85.0  ...    7.2     4.4  14.1   19.2  100.4
6     Lg Rank   NaN     12      3     12  ...      7       6    19      9      3
7   Year/Year   NaN   0.3%  -3.2%  -0.9%  ...  -4.7%  -14.4%  1.6%  -5.6%  -4.7%

[8 rows x 24 columns]

或者

print (tables[18])
    Rk          Unnamed: 1       Salary
0    1      Gordon Hayward  $29,727,900
1    2          Al Horford  $27,734,405
2    3        Kyrie Irving  $18,868,625
3    4        Jayson Tatum   $5,645,400
4    5         Greg Monroe   $5,000,000
5    6       Marcus Morris   $5,000,000
6    7        Jaylen Brown   $4,956,480
7    8        Marcus Smart   $4,538,020
8    9         Aron Baynes   $4,328,000
9   10  Guerschon Yabusele   $2,247,480
10  11        Terry Rozier   $1,988,520
11  12        Shane Larkin   $1,471,382
12  13        Semi Ojeleye   $1,291,892
13  14         Abdel Nader   $1,167,333
14  15        Daniel Theis     $815,615
15  16   Demetrius Jackson      $92,858
16  17        Jarell Eddie      $83,129
17  18        Xavier Silas      $74,159
18  19     Jonathan Gibson      $44,495
19  20         Jabari Bird           $0
20  21        Kadeem Allen           $0

Answer 2

该页面中没有id为team_and_opponent的table。相反，有一个具有此id的span标记。您可以通过更改id来获取结果。

Answer 3

这些数据应该是动态加载的(就像JavaScript一样)。

你应该看看这里Web-scraping JavaScript page with Python

为此，您可以使用支持Javascript的Selenium或html