某些'utf-8‘编解码器无法解码字节

Question

当我使用wget下载网站时，我得到一些错误

代码：

import threading
import urllib.request
import os
import re
import time
import json

def wget(url):
                #self.url = url 
    data = os.popen('wget -qO- %s'% url).read()
    return data

print (wget("http://jamesholm.se/dj.php"))

错误：

Traceback (most recent call last):
  File "stand-alone-check-url.py", line 13, in <module>
    print (wget("http://jamesholm.se/dj.php"))
  File "stand-alone-check-url.py", line 10, in wget
    data = os.popen('wget -qO- %s'% url).read()
  File "/usr/local/lib/python3.4/codecs.py", line 313, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9a in position 13133: invalid start byte

如何克服这个错误？

Answer 1

您不能将任意字节序列解码为utf-8编码文本：

>>> b'\xa9'.decode('utf-8')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa9 in position 0: invalid start byte

该页面指示它使用utf-8，但服务器发送的实际数据不是utf-8。常有的事。

有一个bs4.UnicodeDammit可以让你处理编码不一致的数据：

import bs4 # $ pip install beautifulsoup4

print(bs4.UnicodeDammit.detwingle(b'S\x9aben  - Ostwind Rec').decode('utf-8'))
# -> Sšben  - Ostwind Rec

Answer 2

使用requests python模块而不是wget。

>>> import requests
>>> data = requests.get("http://jamesholm.se/dj.php").text
>>> print(data)