如何使用Python从Met Office JSON下载中提取数据

Question

我正在使用Python3.4。

我已经开始了一个项目，下载英国气象局的预测数据( JSON格式)，并使用这些信息作为我的家庭供暖系统的天气补偿器。我已经成功地从MET Office下载了JSON数据文件，现在我想提取我需要的信息。我可以通过将文件转换为字符串并使用.find和.int方法提取数据来实现这一点，但这看起来很粗糙(但很有效)。由于JSON据说是一种使用良好的数据交换格式，因此必须有更好的方法来做到这一点。我已经找到了像json.load和json.loads这样的东西，也找到了json.JSONDecoder.decode，但是我在使用这些东西方面还没有取得任何成功，而且我真的不知道我在做什么！

我的代码是：

import urllib.request
import json

#Comment:  THIS IS THE CALL TO GET THE MET OFFICE FILE FROM THE INTERNET
#Comment:  **** = my personal met office API key, which I had better keep to myself

response = urllib.request.urlopen('http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/json/354037?res=3hourly&key=****')

FCData    = response.read()
FCDataStr = str(FCData)

#Comment:   END OF THE CALL TO GET MET OFFICE FILE FROM THE INTERNET
#Comment:   Example of data extraction

ChPos = FCDataStr.find('"DV"')      #Find "DV"    
ChPos = FCDataStr.find('"dataDate"', ChPos, ChPos+50)      #Find "dataDate"

FileDataDate = FCDataStr[ChPos+12:ChPos+22]                #Extract the date of the file

#Comment:   And so on

当使用json.loads(FCDataStr)时，我得到以下错误消息：

"ValueError:期望值:第1列(char 0)“

通过删除开头的b‘和结尾的’b‘，这个错误就消失了(参见下面)。使用print(FCDataStr)打印字符串格式的JSON文件提供：

b'{"SiteRep":{"Wx":{"Param":[{"name":"F","units":"C","$":"Feels Like Temperature"},{"name":"G","units":"mph","$":"Wind Gust"},{"name":"H","units":"%","$":"Screen Relative Humidity"},{"name":"T","units":"C","$":"Temperature"},{"name":"V","units":"","$":"Visibility"},{"name":"D","units":"compass","$":"Wind Direction"},{"name":"S","units":"mph","$":"Wind Speed"},{"name":"U","units":"","$":"Max UV Index"},{"name":"W","units":"","$":"Weather Type"},{"name":"Pp","units":"%","$":"Precipitation Probability"}]},"DV":{"dataDate":"2014-07-29T20:00:00Z","type":"Forecast","Location":{"i":"354037","lat":"51.7049","lon":"-2.9022","name":"USK","country":"WALES","continent":"EUROPE","elevation":"43.0","Period":[{"type":"Day","value":"2014-07-29Z","Rep":[{"D":"NNW","F":"22","G":"11","H":"51","Pp":"4","S":"9","T":"24","V":"VG","W":"7","U":"7","$":"900"},{"D":"NW","F":"19","G":"16","H":"61","Pp":"8","S":"11","T":"22","V":"EX","W":"8","U":"1","$":"1080"},{"D":"NW","F":"16","G":"20","H":"70","Pp":"1","S":"11","T":"18","V":"VG","W":"2","U":"0","$":"1260"}]},{"type":"Day","value":"2014-07-30Z","Rep":[{"D":"NW","F":"13","G":"16","H":"84","Pp":"0","S":"7","T":"14","V":"VG","W":"0","U":"0","$":"0"},{"D":"WNW","F":"12","G":"13","H":"90","Pp":"0","S":"7","T":"13","V":"VG","W":"0","U":"0","$":"180"},{"D":"WNW","F":"13","G":"11","H":"87","Pp":"0","S":"7","T":"14","V":"GO","W":"1","U":"1","$":"360"},{"D":"SW","F":"18","G":"9","H":"67","Pp":"0","S":"4","T":"19","V":"VG","W":"1","U":"2","$":"540"},{"D":"WNW","F":"21","G":"13","H":"56","Pp":"0","S":"9","T":"22","V":"VG","W":"3","U":"6","$":"720"},{"D":"W","F":"21","G":"20","H":"55","Pp":"0","S":"11","T":"23","V":"VG","W":"3","U":"6","$":"900"},{"D":"W","F":"18","G":"22","H":"57","Pp":"0","S":"11","T":"21","V":"VG","W":"1","U":"2","$":"1080"},{"D":"WSW","F":"16","G":"13","H":"80","Pp":"0","S":"7","T":"16","V":"VG","W":"0","U":"0","$":"1260"}]},{"type":"Day","value":"2014-07-31Z","Rep":[{"D":"SW","F":"14","G":"11","H":"91","Pp":"0","S":"4","T":"15","V":"GO","W":"0","U":"0","$":"0"},{"D":"SW","F":"14","G":"11","H":"92","Pp":"0","S":"4","T":"14","V":"GO","W":"0","U":"0","$":"180"},{"D":"SW","F":"15","G":"11","H":"89","Pp":"3","S":"7","T":"16","V":"GO","W":"3","U":"1","$":"360"},{"D":"WSW","F":"17","G":"20","H":"79","Pp":"28","S":"11","T":"18","V":"GO","W":"3","U":"2","$":"540"},{"D":"WSW","F":"18","G":"22","H":"72","Pp":"34","S":"11","T":"20","V":"GO","W":"10","U":"5","$":"720"},{"D":"WSW","F":"18","G":"22","H":"66","Pp":"13","S":"11","T":"20","V":"VG","W":"7","U":"5","$":"900"},{"D":"WSW","F":"17","G":"22","H":"69","Pp":"36","S":"11","T":"19","V":"VG","W":"10","U":"2","$":"1080"},{"D":"WSW","F":"16","G":"16","H":"84","Pp":"6","S":"9","T":"17","V":"GO","W":"2","U":"0","$":"1260"}]},{"type":"Day","value":"2014-08-01Z","Rep":[{"D":"SW","F":"16","G":"13","H":"91","Pp":"4","S":"7","T":"16","V":"GO","W":"7","U":"0","$":"0"},{"D":"SW","F":"15","G":"11","H":"93","Pp":"5","S":"7","T":"16","V":"GO","W":"7","U":"0","$":"180"},{"D":"SSW","F":"15","G":"11","H":"93","Pp":"7","S":"7","T":"16","V":"GO","W":"7","U":"1","$":"360"},{"D":"SSW","F":"17","G":"18","H":"79","Pp":"14","S":"9","T":"18","V":"GO","W":"7","U":"2","$":"540"},{"D":"SSW","F":"17","G":"22","H":"74","Pp":"43","S":"11","T":"19","V":"GO","W":"10","U":"5","$":"720"},{"D":"SW","F":"16","G":"22","H":"81","Pp":"48","S":"11","T":"18","V":"GO","W":"10","U":"5","$":"900"},{"D":"SW","F":"16","G":"18","H":"80","Pp":"55","S":"9","T":"17","V":"GO","W":"12","U":"1","$":"1080"},{"D":"SSW","F":"15","G":"16","H":"89","Pp":"38","S":"7","T":"16","V":"GO","W":"9","U":"0","$":"1260"}]},{"type":"Day","value":"2014-08-02Z","Rep":[{"D":"S","F":"14","G":"11","H":"94","Pp":"15","S":"7","T":"15","V":"GO","W":"7","U":"0","$":"0"},{"D":"SSE","F":"14","G":"11","H":"94","Pp":"16","S":"7","T":"15","V":"GO","W":"7","U":"0","$":"180"},{"D":"S","F":"14","G":"13","H":"93","Pp":"36","S":"7","T":"15","V":"GO","W":"10","U":"1","$":"360"},{"D":"S","F":"15","G":"20","H":"84","Pp":"62","S":"11","T":"17","V":"GO","W":"14","U":"2","$":"540"},{"D":"SSW","F":"16","G":"22","H":"78","Pp":"63","S":"11","T":"18","V":"GO","W":"14","U":"5","$":"720"},{"D":"WSW","F":"16","G":"27","H":"66","Pp":"59","S":"13","T":"19","V":"VG","W":"14","U":"5","$":"900"},{"D":"WSW","F":"15","G":"25","H":"68","Pp":"39","S":"13","T":"18","V":"VG","W":"10","U":"2","$":"1080"},{"D":"SW","F":"14","G":"16","H":"80","Pp":"28","S":"9","T":"15","V":"VG","W":"0","U":"0","$":"1260"}]}]}}}}'

使用的结果：

DecodedJSON = json.loads(FCDataStr)
print(DecodedJSON)

给出一个与原始FCDataStr文件非常相似的结果。

如何从文件中提取每3小时预报的数据(如温度、风速等)？

Answer 1

这就是问题所在：

FCDataStr = str(FCData)

当您在一个str对象上调用bytes时，您得到的是bytes对象的字符串表示形式--在引号中，带有b前缀，并带有特殊字符反斜杠--转义。

如果要将二进制数据解码为文本，则必须使用decode方法：

FCDataStr = FCData.decode('utf-8')

(我猜UTF-8是因为除非另有规定，否则JSON总是应该在UTF-8中。)

更详细的是：

urllib.request.urlopen返回一个http.client.HTTPResponse，它是一个类似文件的二进制对象(实现io.RawIOBase)。

您不能将它传递给json.load，因为它需要一个类似文本文件的对象--具有返回str而不是bytes的read方法的东西。您可以将您的HTTPResponse封装在io.BufferedReader中，然后包装而不是在io.TextIOBase中包装(使用encoding='utf-8')，然后将其传递给json.load，但这可能比您想做的工作要多。

所以，最简单的事情就是你想要做的事情，只使用decode而不是str。

data_bytes = response.read() data_str = data_bytes.decode('utf-8') data_dict = json.loads(data_str)

然后，不要尝试访问data_str中的数据--这只是一个字符串，表示数据的JSON编码；data_dict是实际的数据。

例如，要查找dataDate of DV of SiteRep，只需执行以下操作：

data_dict['SiteRep']['DV']['DataDate']

这会让你得到字符串'2014-07-31T14:00:00Z‘。您可能仍然希望将其转换为datetime.datetime对象(因为JSON只理解几种基本类型:字符串、数字、列表和dicts)。但是，它仍然比试图从data_str的find-ing中挑选它或猜测补偿要好得多。

我的猜测是，您已经找到了一些为Python2.x编写的示例代码，您可以通过调用适当的构造函数在字节字符串和Unicode字符串之间进行转换，而无需指定默认为sys.getdefaultencoding()的编码，而且通常(至少在Mac或大多数现代Linux发行版上)是UTF-8，所以尽管错误，它还是工作的。在这种情况下，您可能希望从…中找到一些更好的示例代码来学习。

Answer 2

对于其他可能想使用英国气象局3小时预测数据提要的无知人士，下面是我正在使用的解决方案：

import urllib.request
import json

###  THIS IS THE CALL TO GET THE MET OFFICE FILE FROM THE INTERNET
response = urllib.request.urlopen('http://datapoint.metoffice.gov.uk/public/data/val/wxfcs/all/json/**YourLocationID**?res=3hourly&key=**your_api_key**')
FCData = response.read()
FCDataStr = FCData.decode('utf-8')
###   END OF THE CALL TO GET MET OFFICE FILE FROM THE INTERNET

#Converts JSON data to a dictionary object
FCData_Dic = json.loads(FCDataStr)

#The following are examples of extracting data from the dictionary object.
#The JSON data is heavily nested.
#Each [] goes one level down, usually defined with {} in the JSON data.
dataDate = (FCData_Dic['SiteRep']['DV']['dataDate'])
print('dataDate =',dataDate)

#There are also [] in the JSON data, which are referenced with integers, 
# starting from [0]
#Here, the [0] refers to the first day's block of data defined with [].
DateDay0 = (FCData_Dic['SiteRep']['DV']['Location']['Period'][0]['value'])
print('DateDay0 =',DateDay0)

#The second [0] picks out each of the first day's forecast data, in this case the time, referenced by '$'
TimeOfFC = (FCData_Dic['SiteRep']['DV']['Location']['Period'][0]['Rep'][0]['$'])
print('TimeOfFC =',TimeOfFC)

#Ditto for the temperature.    
Temperature = int((FCData_Dic['SiteRep']['DV']['Location']['Period'][0]['Rep'][0]['T']))
print('Temperature =',Temperature)

#Ditto for the weather Type (a code number).
WeatherType = int((FCData_Dic['SiteRep']['DV']['Location']['Period'][0]['Rep'][0]['W']))
print('WeatherType =',WeatherType)

我希望这能帮上忙！

Answer 3

我一直在解析Met数据池输出。

感谢上面的回应，我有一些对我有用的东西。

我正在将我感兴趣的数据写入CSV文件：

import sys
import os
import urllib.request
import json

###  THIS IS THE CALL TO GET THE MET OFFICE FILE FROM THE INTERNET
response = urllib.request.urlopen('http://datapoint.metoffice.gov.uk/public/data/val/wxobs/all/json/3351?res=hourly&?key=<my key>')
FCData = response.read()
FCDataStr = FCData.decode('utf-8')
###   END OF THE CALL TO GET MET OFFICE FILE FROM THE INTERNET

#Converts JSON data to a dictionary object
FCData_Dic = json.loads(FCDataStr)

# Open output file for appending
fName=<my filename>
if (not os.path.exists(fName)):
    print(fName,' does not exist')
    exit()
fOut=open(fName, 'a')

# Loop through each day, will nearly always be 2 days,
# unless run at midnight. 
i = 0
j = 0
for k in range(24):
    # there will be 24 values altogether
    # find the first hour value for the first day
    DateZ = (FCData_Dic['SiteRep']['DV']['Location']['Period'][i]['value'])
    hhmm = (FCData_Dic['SiteRep']['DV']['Location']['Period'][i]['Rep'][j]  ['$'])
    Temperature = (FCData_Dic['SiteRep']['DV']['Location']['Period'][i]['Rep'][j]['T'])
    Humidity = (FCData_Dic['SiteRep']['DV']['Location']['Period'][i]['Rep'][j]['H'])
    DewPoint = (FCData_Dic['SiteRep']['DV']['Location']['Period'][i]['Rep'][j]['Dp'])
    recordStr = '{},{},{},{},{}\n'.format(DateZ,hhmm,Temperature,Humidity,DewPoint)
    fOut.write(recordStr)
    j = j + 1
    if (hhmm == '1380'):
        i = i + 1
        j = 0
fOut.close()
print('Records added to ',fName)`