Web抓取表没有显示结果

Question

我想从此页上刮走主桌。

​

​

然而，无论是.find_all()还是.xpath()，我都没有得到任何结果。

import requests
from bs4 import BeautifulSoup

page = requests.get('https://paperswithcode.com/sota/3d-object-detection-on-kitti-cars-moderate')
soup = BeautifulSoup(page.text, 'html.parser')
soup.find_all('table')
# out: []

import requests
import lxml.html as lh

page = requests.get('https://paperswithcode.com/sota/3d-object-detection-on-kitti-cars-moderate')
doc = lh.fromstring(page.content)
doc.xpath('//*[@id="leaderboard"]/div[3]/table')
# out: []

有什么问题，我怎样才能得到正确的结果？

Answer 1

因为网页不是动态的，所以您可以在脚本块中找到表。

import requests
import json
from bs4 import BeautifulSoup


url = 'https://paperswithcode.com/sota/3d-object-detection-on-kitti-cars-moderate'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'lxml')
table_json = json.loads(soup.find('script', {'id': 'evaluation-table-data'}).getText())
for row in table_json:
    print(row['rank'], row['method'], row['metrics']['AP'], row['paper']['title'], row['evaluation_date'][:4])

产出：

1 BtcDet 82.86% Behind the Curtain: Learning Occluded Shapes for 3D Object Detection 2021
2 SE-SSD 82.54% SE-SSD: Self-Ensembling Single-Stage Object Detector From Point Cloud 2020
3 SPG 82.13 % SPG: Unsupervised Domain Adaptation for 3D Object Detection via Semantic Point Generation 2021
4 VoTr-TSD (ours) 82.09% Voxel Transformer for 3D Object Detection 2021
5 Pyramid-PV 82.08% Pyramid R-CNN: Towards Better Performance and Adaptability for 3D Object Detection 2021
6 PV-RCNN++ 81.88% PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detection 2021
7 M3DeTR 81.73 M3DeTR: Multi-representation, Multi-scale, Mutual-relation 3D Object Detection with Transformers 2021
8 Voxel R-CNN 81.62% Voxel R-CNN: Towards High Performance Voxel-based 3D Object Detection 2020
9 PV-RCNN 81.43% PV-RCNN: Point-Voxel Feature Set Abstraction for 3D Object Detection 2019
10 SVGA-Net 80.82 % SVGA-Net: Sparse Voxel-Graph Attention Network for 3D Object Detection from Point Clouds 2020
11 CIA-SSD 80.28% CIA-SSD: Confident IoU-Aware Single-Stage Object Detector From Point Cloud 2020
12 SA-SSD+EBM 80.12% Accurate 3D Object Detection using Energy-Based Models 2020
13 PC-RGNN 79.9% PC-RGNN: Point Cloud Completion and Graph Neural Network for 3D Object Detection 2020
14 Joint 78.96% Joint 3D Instance Segmentation and Object Detection for Autonomous Driving 2020
15 STD 77.63% STD: Sparse-to-Dense 3D Object Detector for Point Cloud 2019
16 UberATG-MMF 76.75% Multi-Task Multi-Sensor Fusion for 3D Object Detection 2020
17 F-ConvNet 76.51% Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection 2019
18 PointRGCN 75.73% PointRGCN: Graph Convolution Networks for 3D Vehicles Detection Refinement 2019
19 PointRCNN 75.42% PointRCNN: 3D Object Proposal Generation and Detection from Point Cloud 2018
20 PointPillars 74.99% PointPillars: Fast Encoders for Object Detection from Point Clouds 2018
21 PC-CNN-V2 73.80% A General Pipeline for 3D Detection of Vehicles 2018
22 RoarNet 73.04% RoarNet: A Robust 3D Object Detection based on RegiOn Approximation Refinement 2018
23 3D-FCT 72.79% 3D-FCT: Simultaneous 3D Object Detection and Tracking Using Feature Correlation 2021
24 IPOD 72.57% IPOD: Intensive Point-based Object Detector for Point Cloud 2018
25 AVOD + Feature Pyramid 71.88% Joint 3D Proposal Generation and Object Detection from View Aggregation 2017
26 Frustum PointNets  70.39% Frustum PointNets for 3D Object Detection from RGB-D Data 2017
27 VoxelNet 65.11% VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection 2017
28 PGD 11.77% Probabilistic and Geometric Depth: Detecting Objects in Perspective 2021