使用lxml分析url html时，无法加载外部实体

Question

我正在尝试解析提供的url中的表情符号。这是Data Wrangling中的一个教程，帮助我理解数据分析。这是课本中的一段逐字代码，最后给出了错误。我听说过使用urllib2的建议，但本练习的重点是使用lxml。这是可以实现的吗，还是这个例子已经过时了？你能提供一些洞察力让它运行，解析表情符号，然后返回列表吗？

from lxml import html

page = html.parse('http://www.emoji-cheat-sheet.com/')

proper_headers = page.xpath('//h2|//h3')
proper_lists = page.xpath('//ul')

all_emoji = []

for header, list_cont in zip(proper_headers, proper_lists):
    section = header.text
    for li in list_cont.getchildren():
        emoji_dict = {}
        spans = li.xpath('div/span')
        if len(spans):
            link = spans[0].get('data-src')
            if link:
                emoji_dict['emoji_link'] = li.base_url + link
            else:
                emoji_dict['emoji_link'] = None
            emoji_dict['emoji_handle'] = spans[1].text_content()
        else:
            emoji_dict['emoji_link'] = None
            emoji_dict['emoji_handle'] = li.xpath('div')[0].text_content()
        emoji_dict['section'] = section
        all_emoji.append(emoji_dict)

print all_emoji

错误：

Traceback (most recent call last):
  File "chp11-scraping/lxml_emoji_xpath.py", line 24, in <module>
    page = html.parse('http://www.emoji-cheat-sheet.com/')
  File "/home/ryan/.local/lib/python2.7/site-packages/lxml/html/__init__.py", line 940, in parse
    return etree.parse(filename_or_url, parser, base_url=base_url, **kw)
  File "src/lxml/etree.pyx", line 3426, in lxml.etree.parse
  File "src/lxml/parser.pxi", line 1840, in lxml.etree._parseDocument
  File "src/lxml/parser.pxi", line 1866, in lxml.etree._parseDocumentFromURL
  File "src/lxml/parser.pxi", line 1770, in lxml.etree._parseDocFromFile
  File "src/lxml/parser.pxi", line 1163, in lxml.etree._BaseParser._parseDocFromFile
  File "src/lxml/parser.pxi", line 601, in lxml.etree._ParserContext._handleParseResultDoc
  File "src/lxml/parser.pxi", line 711, in lxml.etree._handleParseResult
  File "src/lxml/parser.pxi", line 638, in lxml.etree._raiseParseError
IOError: Error reading file 'http://www.emoji-cheat-sheet.com/': failed to load external entity "http://www.emoji-cheat-sheet.com/"

Answer 1

它重定向到不支持的https和lxml，请使用urllib2或requests读取html

from lxml import html
import urllib2

# https://www.webpagefx.com/tools/emoji-cheat-sheet/
doc = urllib2.urlopen('http://www.emoji-cheat-sheet.com/')
page = html.parse(doc)