用ElementTree和BeautifulSoup解析文件:是否有一种方法可以按标记级别的数量来解析文件？

Question

我有这 xml文件，我基本上希望将所有信息记录到字典中。

我写了这段代码：

import requests
import xml.etree.ElementTree as ET
import urllib2
import glob
import pprint
from bs4 import BeautifulSoup

#get the XML file
#response = requests.get('https://www.drugbank.ca/drugs/DB01048.xml')
#with open('output.txt', 'w') as input:
#          input.write(response.content)


#set up lists etc
set_of_files = glob.glob('output*txt')
val = lambda x: "{http://www.drugbank.ca}" + str(x)
key_list = ['drugbank-id','name','description','cas-number','unii','average-mass','monoisotopic-mass','state','indication','pharmacodynamics','mechanism-of-action','toxicity','metabolism','absorption','half-life','protein-binding','route-of-elimination','volume-of-distribution','fda-label','msds']
key_dict = {}
method1 = ['groups','synonyms','patent']
method3_inputs = [('dosages','dosage'),('salts','salt'),('products','product'),('mixtures','mixture'),('packagers','packager'),('categories','category'),('atc-codes','atc-code'),('pdb-entries','pdb-entry'),('food-interactions','food-interaction'),('drug-interactions','drug-interaction'),('properties','property'),('external-identifiers','external-identifier'),('external-links','external-link'),('reactions','reaction')]
list_to_run_thru = ['description','direct-parent','kingdom','superclass','class','subclass']
alternative_parents = []
http_add = '{http://www.drugbank.ca}'
substituents = []
ap_sub = lambda x:'{http://www.drugbank.ca}'+ x

def method2(list2_name,list3_name,list4_name):
     temp_list = []
     for i in subnode:
          if i.tag == list2_name:
               for a in i:
                    if a.tag == list3_name:
                         for u in a:
                              if u.tag == list4_name:
                                   temp_list.append(u.text)
     return temp_list

def method3(list1_name):
     list_of_tuples = []
     for i in subnode:
          if i.tag == list1_name:
               temp_list = []
               for a in i:
                    temp_list.append(a.text)
                    list_of_tuples.append(temp_list)
     return list_of_tuples

new_method3 = []
for i in method3_inputs:
     new_k = http_add + i[0]
     new_v = http_add + i[1]
     new_method3.append((new_k,new_v))

for each_file in set_of_files:
     tree = ET.parse(each_file)
     root = tree.getroot()
     for i in key_list:
          for child in root.getchildren():
               if i not in key_dict:
                    key_dict[i] = [child.find(val(i)).text.encode('utf-8')]
               else:
                    key_dict[i].append(child.find(val(i)).text.encode('utf-8'))

     for node in root:
          for subnode in node:
               if subnode.tag == '{http://www.drugbank.ca}general-references':
                    if 'pubmed-id' not in key_dict:
                         key_dict['pubmed-id'] = method2('{http://www.drugbank.ca}articles','{http://www.drugbank.ca}article','{http://www.drugbank.ca}pubmed-id')
                    else:
                         key_dict['pubmed-id'].append(method2('{http://www.drugbank.ca}articles','{http://www.drugbank.ca}article','{http://www.drugbank.ca}pubmed-id'))

               for a,b in zip(method3_inputs,new_method3):
                    if subnode.tag == b[0]:
                         if a[0] not in key_dict:
                              key_dict[a[0]] = method3(b[1])
                         else:
                              key_dict[method3_inputs].append(method3(b[1]))

               if subnode.tag == '{http://www.drugbank.ca}classification': 
                    for each_item in list_to_run_thru:
                         for i in subnode:
                               if i.tag == ap_sub(each_item):
                                   if i.tag == '{http://www.drugbank.ca}alternative-parent':
                                         alternative_parents.append(i.text)  
                                   if i.tag == '{http://www.drugbank.ca}substituent':
                                         substituents.append(i.text)  

               for i in method1:
                    if subnode.tag == '{http://www.drugbank.ca}' + i:
                         for n in subnode:
                              if i not in key_dict:
                                   key_dict[i] = [n.text]
                              elif i in key_dict:
                                   key_dict[i].append(n.text)

               if subnode.tag == '{http://www.drugbank.ca}pathways':
                    if 'pathways'not in key_dict:
                         key_dict['pathways'] = method2('{http://www.drugbank.ca}pathways','{http://www.drugbank.ca}pathway','{http://www.drugbank.ca}drug')
                    else:
                         key_dict['pathways'].append(method2('{http://www.drugbank.ca}pathways','{http://www.drugbank.ca}pathway','{http://www.drugbank.ca}drug'))


key_dict['alternative_parents'] = alternative_parents
key_dict['substituent'] = substituents


html = requests.get('https://www.drugbank.ca/drugs/DB01048').text
soup = BeautifulSoup(html, 'html.parser')
div_targets = soup.find('div', class_='bond-list-container targets')
targets = div_targets.find_all('div', class_='bond card')

for target in targets:
    k = []
    v = []
    for property in target.find_all('dt'):
        k.append(property.get_text())
    for property in target.find_all('dd'):
        v.append(property.get_text())
    key_dict[target.find('strong').get_text()] = dict(zip(k, v))

print key_dict.keys()

代码运行在描述良好的XML文件上。但有一些问题我想看看是否有人可以改进：

1. 你可以清楚地看到我不得不对它进行大量的硬编码。例如，一些标签是一层深的(例如drugbank-id)，有些是两层深的(例如组->组)，有些是三层深的(例如产品、产品、名称)等等。

您可以看到，我已经为两层和三层深度(函数称为method3和method2)编写了一个函数，然后我在属于每个函数的列表中进行了硬编码。

我想知道有谁能告诉我比这更好/更干净/更高效的代码吗？我的想法是，我有一个is列表(例如描述、产品)，而函数不知何故理解‘这是一个3层标签(例如products -> products ->name)，所以我不必在所有3层标签、所有2层标签中硬编码等等，比如有一个if语句'if标签是3层的deep...do -deep...do function....if标记是2层的- deep....do this’。

1. 免责声明:我知道这个方法同时使用BeautifulSoup和ET来解析，因为我被困在了一个部分并得到了帮助这里。不幸的是，我需要掌握一个，然后才能继续前进，我发现HTML版本比XML要混乱得多，所以这就是为什么脚本从XML和HTML的角度解析文件的原因。
2. 有没有人对如何使这个脚本变得更干净有任何一般性的评论？这样做的想法是把每个“顶级”标签(即“毒品类型”标签下面的所有东西)，然后基本上把所有的信息放到字典/列表/其他什么地方，然后每当我想搜索什么东西的时候，我就可以搜索字典中的“顶级”单词，它有我可以读取的标签的子数据(如果标签下面只有一个级别，那么只要一个字符串就可以了，但是如果标签下面有很多标签，可能会返回字典/列表/元组/更合适的东西)。

编辑:基于下面的帮助，我下载了drugbank.xsd并运行了以下命令：

pyxbgen -m DB01048 -u drugbank.xsd --no-validate-changes

然后这个剧本：

from __future__ import print_function
import DB01048

xml = open('DB01048.xml').read()
d_01048 = DB01048.CreateFromDocument(xml)

#print(d_01048.drug[0].state)
#print(d_01048.drug[0].name)
#print(d_01048.drug[0].general_references.articles.article[0].pubmed_id)

而错误是：

Traceback (most recent call last):
  File "parse_drug_db.py", line 5, in <module>
    d_01048 = DB01048.CreateFromDocument(xml)
  File "/home/drugs/DB01048.py", line 65, in CreateFromDocument
    saxer.parse(io.BytesIO(xmld))
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 110, in parse
    xmlreader.IncrementalParser.parse(self, source)
  File "/usr/lib/python2.7/xml/sax/xmlreader.py", line 123, in parse
    self.feed(buffer)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 213, in feed
    self._parser.Parse(data, isFinal)
  File "/usr/lib/python2.7/xml/sax/expatreader.py", line 365, in end_element_ns
    self._cont_handler.endElementNS(pair, None)
  File "/home/aoidoh/bin/local/lib/python2.7/site-packages/pyxb/binding/saxer.py", line 388, in endElementNS
    binding_object = this_state.endBindingElement()
  File "/home/aoidoh/bin/local/lib/python2.7/site-packages/pyxb/binding/saxer.py", line 245, in endBindingElement
    return self.__bindingInstance._postDOMValidate()
  File "/home/aoidoh/bin/local/lib/python2.7/site-packages/pyxb/binding/basis.py", line 2652, in _postDOMValidate
    self._validateAttributes()
  File "/home/aoidoh/bin/local/lib/python2.7/site-packages/pyxb/binding/basis.py", line 2246, in _validateAttributes
    au.validate(self)
  File "/home/aoidoh/bin/local/lib/python2.7/site-packages/pyxb/binding/content.py", line 251, in validate
    raise pyxb.MissingAttributeError(type(ctd_instance), self.__name, ctd_instance)
pyxb.exceptions_.MissingAttributeError: Instance of <class 'DB01048.drugbank_type'> lacks required attribute exported-on

Answer 1

更多的pythonic方法可能是使用PyXB包来生成从xml文件中解封的python对象。

• 安装PyXB软件包

pip install PyXB

• 从下载的xsd创建python模块。将创建DB01048.py

pyxbgen -m DB01048 -u drugbank-plus.xsd --no-validate-changes

• 将生成的模块用作 从print(d_01048.drug.general_references.articles.article.pubmed_id)导入print_function DB01048 = open('DB01048.xml').read() d_01048 = DB01048.CreateFromDocument( xml ) print(d_01048.drug.state) print(d_01048.drug.name)

solid Abacavir 17356469

如果出现与某些MissingAttribute属性相关的exported-on错误，请在on 01048.py上的第一个类之前添加以下行

pyxb.RequireValidWhenParsing(True) pyxb.RequireValidWhenParsing(False)