Python urllib.request和urllib.error用法

Question

我有少量的Python代码，它们使用urllib.request和urllib.error来检查URL的状态。如果出现错误，这将触发SNS发布到主题。

import boto3
from urllib.request import urlopen
from urllib.error import HTTPError
from os import environ

region = 'eu-west-2'

def get_status_code(url, topic):
    try:
        connection = urlopen(url, timeout=10)
        status = connection.getcode()
        print('%s Status is %s' % (url, status))
        connection.close()
    except HTTPError as err:
        status = err.getcode()
        print('%s: Status is unreachable - %s' % (url, status))
        sns_client = boto3.client('sns', region_name=region)
        message = sns_client.publish(TopicArn=topic,
                                     Message='%s is unreachable - HTTP error code is: %s' % (url, status)
                                     )
        print("Publish to SNS, Lambda update triggered: {}".format(message))

def lambda_handler(event, context):
    dns = environ['dns']
    sns_topic = environ['sns_topic_arn']

    get_status_code(dns, sns_topic)

作为测试的一部分，我经历了不同程度的成功。如果我将超时保存在urlopen中，它将无法触发except处理程序，但会转储一个失败。如果设置超时限制并且命中被认为是错误，那么为什么不触发异常处理程序呢？

timed out: timeout
Traceback (most recent call last):
  File "/var/task/lambda_function1.py", line 27, in lambda_handler
    get_status_code(elb_dns, sns_topic)
  File "/var/task/lambda_function1.py", line 10, in get_status_code
    connection = urlopen(url, timeout=10)
  File "/var/lang/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/var/lang/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/var/lang/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/var/lang/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/var/lang/lib/python3.6/urllib/request.py", line 1377, in http_open
    return self.do_open(http.client.HTTPConnection, req)
  File "/var/lang/lib/python3.6/urllib/request.py", line 1352, in do_open
    r = h.getresponse()
  File "/var/lang/lib/python3.6/http/client.py", line 1379, in getresponse
    response.begin()
  File "/var/lang/lib/python3.6/http/client.py", line 311, in begin
    version, status, reason = self._read_status()
  File "/var/lang/lib/python3.6/http/client.py", line 272, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/var/lang/lib/python3.6/socket.py", line 586, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

如果我从urlopen中删除超时，作业将在3分钟后超时，并触发except处理程序，然后成功完成(我知道如果它在10秒后仍未工作，它将不会工作)。

如果我使用HTTPS URL，还会出现第二个意外错误。它也无法触发异常处理程序。

<urlopen error timed out>: URLError
Traceback (most recent call last):
  File "/var/task/lambda_function1.py", line 24, in lambda_handler
    get_status_code(elb_dns, sns_topic)
  File "/var/task/lambda_function1.py", line 8, in get_status_code
    connection = urlopen(url, timeout=10)
  File "/var/lang/lib/python3.6/urllib/request.py", line 223, in urlopen
    return opener.open(url, data, timeout)
  File "/var/lang/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/var/lang/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/var/lang/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/var/lang/lib/python3.6/urllib/request.py", line 1392, in https_open
    context=self._context, check_hostname=self._check_hostname)
  File "/var/lang/lib/python3.6/urllib/request.py", line 1351, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error timed out>

Answer 1

因此，当我添加输出的错误时，我注意到它们没有写入HTTPError (我预计超时至少在默认超时写入的情况下是？)异常处理程序所期望的：

import boto3
import socket
from socket import AF_INET, SOCK_DGRAM
from urllib.request import urlopen
from urllib.error import HTTPError
from urllib.error import URLError
from os import environ

region = 'eu-west-2'

def get_status_code(url, topic):
    try:
        connection = urlopen(url, timeout=10)
        status = connection.getcode()
        print('%s Status is %s' % (url, status))
        connection.close()
    except (socket.timeout, URLError, HTTPError) as err:
        status = err
        print('%s: Status is unreachable - %s' % (url, status))
        sns_client = boto3.client('sns', region_name=region)
        message = sns_client.publish(TopicArn=topic,
                                     Message='%s is unreachable - HTTP error code is: %s' % (url, status)
                                     )
        print("Publish to SNS, Lambda update triggered: {}".format(message))

def lambda_handler(event, context):
    dns = environ['dns']
    sns_topic = environ['sns_topic_arn']

    get_status_code(dns, sns_topic)

因此，我将新错误添加到异常处理程序中，现在正在捕获和处理这些错误。它确实打破了我的"HTTP错误代码是“，因为我现在只是得到了错误消息，但它比以前工作得更好。