向下滚动网站时，Python抓取会自动加载后续的新闻页面

Question

当向下滚动财经新闻网站http://www.aastocks.com/tc/stocks/news/aafn-company-news时，它会自动加载下一页。我用Chrome>Inspect>Network查看了网站。我发现为了获取更多的新闻，请求地址是"http://www.aastocks.com/tc/resources/datafeed/getmorenews.ashx?cat=company-news&newstime=975758621&newsid=NOW.1060532&period=0&key=&symbol=“，请求方法是" get”。我检查了网络中的RESPONSE选项卡，数据是JSON格式的。以下代码不起作用。是不是少了点什么？

url ="http://www.aastocks.com/tc/resources/datafeed/getmorenews.ashx?cat=company-news&newstime=975758615&newsid=NOW.1060527&period=0&key=&symbol="
df = pd.read_json(url) 

Answer 1

您需要设置referer标头。因此，一种简单的方法是使用requests来获取json。并使用pandas.DataFrame将其转换为下面的directly.Try代码：

import requests
import pandas as pd
headers = {
    'Referer': 'http://www.aastocks.com/tc/stocks/news/aafn-company-news',
}

params = (
    ('cat', 'company-news'),
    ('newstime', '975758615'),
    ('newsid', 'NOW.1060527'),
    ('period', '0'),
    ('key', ''),
    ('symbol', ''),
)

response = requests.get('http://www.aastocks.com/tc/resources/datafeed/getmorenews.ashx', headers=headers, params=params)
df = pd.DataFrame(response.json())
print(df)

结果：

    t                 id                dt  ... becnt rcnt cv
0   1  .HK.201202_115703  2020/12/02 11:57  ...     3    5  1
1   1  .HK.201202_115445  2020/12/02 11:54  ...    81   96  1
2   1        NOW.1060528  2020/12/02 11:47  ...   129  116  1
3   1        NOW.1060522  2020/12/02 11:47  ...     4    5  1
4   1        NOW.1060523  2020/12/02 11:47  ...     8   15  1
5   1        NOW.1060520  2020/12/02 11:43  ...   159   53  1
6   1        NOW.1060518  2020/12/02 11:37  ...     2    0  1
7   1        NOW.1060517  2020/12/02 11:35  ...    33   49  1
8   1  RUM.201202_113518  2020/12/02 11:35  ...    65   52  1
9   1        NOW.1060514  2020/12/02 11:34  ...     0    0  1
10  1  RUM.201202_112609  2020/12/02 11:26  ...    16   17  1
11  1        NOW.1060510  2020/12/02 11:21  ...    11   26  1
12  1        NOW.1060508  2020/12/02 11:14  ...    42   58  1
13  1        NOW.1060506  2020/12/02 11:12  ...     4    2  1
14  1        NOW.1060502  2020/12/02 11:12  ...     0    0  1
15  1        NOW.1060507  2020/12/02 10:59  ...     9    3  1
16  1        NOW.1060497  2020/12/02 10:56  ...     2    0  1
17  1        NOW.1060496  2020/12/02 10:51  ...     2    1  1
18  1        NOW.1060495  2020/12/02 10:47  ...     0    0  1
19  1        NOW.1060492  2020/12/02 10:43  ...     0    0  1

[20 rows x 12 columns]