python urllib2超时

Question

好了，伙计们，我已经在谷歌和这里的stackoverflow中搜索了这个答案，几个小时后，我没有看到一个正确的答案来做这个工作的脚本……

在这里，我粘贴了4个假定的python工作脚本示例，用于为不存在的url设置默认超时，并使用套接字和/或超时参数设置超时。

没有人工作，所以永远不会触发超时。

有什么想法吗？

第一个示例：

import urllib2

try:                
    header_s = {"User-Agent":"Mozilla/5.0 (X11; U; Linux i686) Gecko/20071127 Firefox/2.0.0.11"}

    req = urllib2.Request("http://www.nonexistantdomainurl.com/notexist.php",headers = header_s)


    print urllib2.urlopen(req, None, 5.0).read()

except urllib2.URLError, e:
    print "Url Error: %r" % e

except Exception,e:
  print "Fallo de tipo ",e

else: 
    print "all ok!"

第二个例子：

import urllib2

try:
    response = urllib2.urlopen("http://www.nonexistantdomainurl.com/notexist.php", None, 2.5)
except urllib2.URLError, e:
    print "Oops, timed out?"

第三个例子：

from urllib2 import Request, urlopen, URLError, HTTPError
import base64


req = Request('http://www.nonexistantdomainurl.com/notexist.php')

try:
    response = urlopen(req,timeout=5.0)   

except HTTPError, e:
    print 'The server couldn\'t fulfill the request.'
    print 'Error code: ', e.code
except URLError, e:
    print 'We failed to reach a server.'
    print 'Reason: ', e.reason

第四个示例：

import urllib2
import socket


socket.setdefaulttimeout(5)

try:
    response = urllib2.urlopen("http://www.faluquito.com/equipo.php",timeout=5.0).read()   


except urllib2.URLError, e:
    print "Url Error: %r" % e

Answer 1

>>> import urllib2
>>> import time
>>> import contextlib
>>>
>>> def timeit():
...   s = time.time()
...   try:
...     yield
...   except urllib2.URLError:
...     pass
...   print 'took %.3f secs' % (time.time() - s)
...
>>> timeit = contextlib.contextmanager(timeit)
>>> with timeit():
...   r = urllib2.urlopen('http://loc:8080', None, 2)
...
took 2.002 secs
>>> with timeit():
...   r = urllib2.urlopen('http://loc:8080', None, 5)
...
took 5.003 secs

Answer 2

如果您的计算机具有unix程序摘要，您可能能够识别不存在的urls，如下所示：

import logging
import subprocess
import shlex

logging.basicConfig(level = logging.DEBUG,
                    format = '%(asctime)s %(module)s %(levelname)s: %(message)s',
                    datefmt = '%M:%S')
logger = logging.getLogger(__name__)

urls = ['http://1.2.3.4',
       "http://www.nonexistantdomainurl.com/notexist.php",
       "http://www.faluquito.com/equipo.php",
        'google.com']

nonexistent = ['63.251.179.13', '8.15.7.117']
for url in urls:
    logger.info('Trying {u}'.format(u=url))

    proc = subprocess.Popen(shlex.split(
        'dig +short +time=1 +retry=0 {u}'.format(u = url)),
                            stdout = subprocess.PIPE, stderr = subprocess.PIPE)
    out, err = proc.communicate()
    out = out.splitlines()
    logger.info(out)
    if any(addr in nonexistent for addr in out):
        logger.info('nonexistent\n')
    else:
        logger.info('success\n')

在我的机器上，这会产生以下结果：

00:57 test INFO: Trying http://1.2.3.4
00:58 test INFO: ['63.251.179.13', '8.15.7.117']
00:58 test INFO: nonexistent

00:58 test INFO: Trying http://www.nonexistantdomainurl.com/notexist.php
00:58 test INFO: ['63.251.179.13', '8.15.7.117']
00:58 test INFO: nonexistent

00:58 test INFO: Trying http://www.faluquito.com/equipo.php
00:58 test INFO: ['63.251.179.13', '8.15.7.117']
00:58 test INFO: nonexistent

00:58 test INFO: Trying google.com
00:58 test INFO: ['72.14.204.113', '72.14.204.100', '72.14.204.138', '72.14.204.102', '72.14.204.101']
00:58 test INFO: success

请注意，对于不存在的urls，dig返回['63.251.179.13', '8.15.7.117']。

我认为我的ISP正在将不存在的地址更改为63.251.179.13或8.15.7.117。您的ISP可能会做一些不同的事情。在这种情况下，您可能需要将nonexistent更改为其他值。