如何从investing.com中抓取数据？

Question

我想从https://au.investing.com/currencies/eur-usd上摘录欧元/美元的5分钟技术总结，但我不知道该怎么做。我试过使用requests模块，但它显示我被禁止访问该网站。

Answer 1

服务器从外部URL加载数据。您可以使用requests模块来加载它。例如，打印candles/times：

import json
import datetime
import requests 
from bs4 import BeautifulSoup


url = 'https://au.investing.com/common/modules/js_instrument_chart/api/data.php?pair_id=1&pair_id_for_news=1&chart_type=area&pair_interval=300&candle_count=120&events=yes&volume_series=yes&period='
headers = {
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:81.0) Gecko/20100101 Firefox/81.0',
    'X-Requested-With': 'XMLHttpRequest',
    'Referer': 'https://au.investing.com/currencies/eur-usd'}
data = requests.get(url, headers=headers).json()

# uncomment this to print all data:
# print( json.dumps(data, indent=4) )

for candle in data['candles']:
    t = datetime.datetime.fromtimestamp(candle[0] // 1000)
    print('{!s:<20} {:<10} {:<10} {:<10}'.format(t, *candle[1:]))

打印：

2020-10-21 21:40:00  1.1859     123        1004      
2020-10-21 21:45:00  1.1859     184        1127      
2020-10-21 21:50:00  1.1862     173        1311      
2020-10-21 21:55:00  1.186      174        1484      
2020-10-21 22:00:00  1.1863     291        1658      
2020-10-21 22:05:00  1.1864     226        1949      
2020-10-21 22:10:00  1.1863     197        2175      
2020-10-21 22:15:00  1.1863     112        2372      
2020-10-21 22:20:00  1.1863     149        2484      
2020-10-21 22:25:00  1.1862     93         2633      
2020-10-21 22:30:00  1.1863     111        2726      
2020-10-21 22:35:00  1.1861     121        2837      
2020-10-21 22:40:00  1.1862     151        2958      
2020-10-21 22:45:00  1.1861     34         3109      
2020-10-21 22:50:00  1.1861     236        3143      
2020-10-21 22:55:00  1.186      148        3379      
2020-10-21 23:00:00  1.1861     202        3527      
2020-10-21 23:05:00  1.1859     264        3729      
2020-10-21 23:10:00  1.186      162        3993      
2020-10-21 23:15:00  1.1859     223        4155      

...and so on.