Scraperwiki字符编码异常

Question

这是一个用Python语言编写的ScraperWiki抓取器：

import lxml.html
import scraperwiki
from unidecode import unidecode

html = scraperwiki.scrape("http://www.timeshighereducation.co.uk/world-university-rankings/2012-13/world-ranking/range/001-200")
root = lxml.html.fromstring(html)
for tr in root.cssselect("table.ranking tr"):
    if len(tr.cssselect("td.rank")) > 0 and len(tr.cssselect("td.uni")) > 0:
        university = unidecode(tr.cssselect("td.uni")[0].text_content()).strip().title()
        if 'cole' in university:
            print university

它会产生以下输出：

Ecole Polytechnique Federale De Lausanne
Ecole Normale Superieure
Acole Polytechnique
Ecole Normale Superieure De Lyon

我的问题是:是什么导致第三个输出行的初始字符呈现为"A“而不是"E"，我如何才能阻止这种情况的发生？

Answer 1

基于soulseekah上面有用的注释，以及lxml文档here和here，下面的解决方案是可行的：

import lxml.html
import scraperwiki
from unidecode import unidecode
from BeautifulSoup import UnicodeDammit

def decode_html(html_string):
    converted = UnicodeDammit(html_string, isHTML=True)
    if not converted.unicode:
        raise UnicodeDecodeError(
            "Failed to detect encoding, tried [%s]",
            ', '.join(converted.triedEncodings))
    return converted.unicode

html = scraperwiki.scrape("http://www.timeshighereducation.co.uk/world-university-rankings/2012-13/world-ranking/range/001-200")
root = lxml.html.fromstring(decode_html(html))
for tr in root.cssselect("table.ranking tr"):
    if len(tr.cssselect("td.rank")) > 0 and len(tr.cssselect("td.uni")) > 0:
        university = unidecode(tr.cssselect("td.uni")[0].text_content()).strip().title()
        if 'cole' in university:
            print university