WGET - HTTPS vs HTTP = HTTPS慢速

Question

我正在尝试使用wget进行一些测试，并且我注意到在wget中加载HTTPS页面比在同一台服务器上加载http要重要。这似乎与任何网络差异无关。在名称解析之前，wget需要大约5秒的时间。有人能帮忙吗？我怎样才能克服这一切？当我注意到这一点时，我正在寻找使用-p和-H选项的wget来评估网络性能。

xbian@xbian ~ $ wget -V
GNU Wget 1.13.4 built on linux-gnueabihf.

+digest +https +ipv6 +iri +large-file +nls -ntlm +opie +ssl/gnutls

Wgetrc:
    /etc/wgetrc (system)
Locale: /usr/share/locale
Compile: gcc -DHAVE_CONFIG_H -DSYSTEM_WGETRC="/etc/wgetrc"
    -DLOCALEDIR="/usr/share/locale" -I. -I../lib -I../lib
    -D_FORTIFY_SOURCE=2 -Iyes/include -g -O2 -fstack-protector
    --param=ssp-buffer-size=4 -Wformat -Werror=format-security
    -DNO_SSLv2 -D_FILE_OFFSET_BITS=64 -g -Wall
Link: gcc -g -O2 -fstack-protector --param=ssp-buffer-size=4 -Wformat
    -Werror=format-security -DNO_SSLv2 -D_FILE_OFFSET_BITS=64 -g -Wall
    -Wl,-z,relro -Lyes/lib -lgnutls -lgcrypt -lgpg-error -lz -lidn -lrt
    ftp-opie.o gnutls.o ../lib/libgnu.a

Copyright (C) 2009 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later
<http://www.gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Originally written by Hrvoje Niksic <hniksic@xemacs.org>.
Please send bug reports and questions to <bug-wget@gnu.org>.
xbian@xbian ~ $ time wget -d -v --no-check-certificate --delete-after -4 http://www.google.pt 2>&1  | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'
2015-02-07 01:10:57 Setting --verbose (verbose) to 1
2015-02-07 01:10:57 Setting --check-certificate (checkcertificate) to 0
2015-02-07 01:10:57 Setting --delete-after (deleteafter) to 1
2015-02-07 01:10:57 Setting --inet4-only (inet4only) to 1
2015-02-07 01:10:57 DEBUG output created by Wget 1.13.4 on linux-gnueabihf.
2015-02-07 01:10:57
2015-02-07 01:10:57 URI encoding = `UTF-8'
2015-02-07 01:10:57 --2015-02-07 01:10:57--  http://www.google.pt/
2015-02-07 01:10:57 Resolving www.google.pt (www.google.pt)... 213.30.5.52, 213.30.5.24, 213.30.5.18, ...
2015-02-07 01:10:57 Caching www.google.pt => 213.30.5.52 213.30.5.24 213.30.5.18 213.30.5.25 213.30.5.59 213.30.5.31 213.30.5.45 213.30.5.46 213.30.5.39 213.30.5.53 213.30.5.32 213.30.5.38
2015-02-07 01:10:57 Connecting to www.google.pt (www.google.pt)|213.30.5.52|:80... connected.
2015-02-07 01:10:57 Created socket 3.
2015-02-07 01:10:57 Releasing 0x003b8040 (new refcount 1).
2015-02-07 01:10:57
2015-02-07 01:10:57 ---request begin---
2015-02-07 01:10:57 GET / HTTP/1.1
2015-02-07 01:10:57 User-Agent: Wget/1.13.4 (linux-gnueabihf)
2015-02-07 01:10:57 Accept: */*
2015-02-07 01:10:57 Host: www.google.pt
2015-02-07 01:10:57 Connection: Keep-Alive
2015-02-07 01:10:57
2015-02-07 01:10:57 ---request end---
2015-02-07 01:10:58 HTTP request sent, awaiting response...
2015-02-07 01:10:58 ---response begin---
2015-02-07 01:10:58 HTTP/1.1 200 OK
2015-02-07 01:10:58 Date: Sat, 07 Feb 2015 01:10:58 GMT
2015-02-07 01:10:58 Expires: -1
2015-02-07 01:10:58 Cache-Control: private, max-age=0
2015-02-07 01:10:58 Content-Type: text/html; charset=ISO-8859-1
2015-02-07 01:10:58 Set-Cookie: PREF=ID=98608883e4031983:FF=0:TM=1423271458:LM=1423271458:S=BnwaLDxFbjCUyPnF; expires=Mon, 06-Feb-2017 01:10:58 GMT; path=/; domain=.google.pt
2015-02-07 01:10:58 Set-Cookie: NID=67=AkXpY2nJPDDcH7xKJkslxdCtflnhOZJiNwZdu4YBAIc2FnjIZIAYHzFuln5boxiOHq1WWBdbcTnLXwPqOrfxOxkLXtO2U5UAVBCU0nVcgyC61_YLZLXGR0Fmdi9M_fIp; expires=Sun, 09-Aug-2015 01:10:58 GMT; path=/; domain=.google.pt; HttpOnly
2015-02-07 01:10:58 P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
2015-02-07 01:10:58 Server: gws
2015-02-07 01:10:58 X-XSS-Protection: 1; mode=block
2015-02-07 01:10:58 X-Frame-Options: SAMEORIGIN
2015-02-07 01:10:58 Alternate-Protocol: 80:quic,p=0.02
2015-02-07 01:10:58 Accept-Ranges: none
2015-02-07 01:10:58 Vary: Accept-Encoding
2015-02-07 01:10:58 Transfer-Encoding: chunked
2015-02-07 01:10:58
2015-02-07 01:10:58 ---response end---
2015-02-07 01:10:58 200 OK
2015-02-07 01:10:58 cdm: 1 2 3 4 5 6 7 8
2015-02-07 01:10:58 Stored cookie google.pt -1 (ANY) / <permanent> <insecure> [expiry 2017-02-06 01:10:58] PREF ID=98608883e4031983:FF=0:TM=1423271458:LM=1423271458:S=BnwaLDxFbjCUyPnF
2015-02-07 01:10:58 cdm: 1 2 3 4 5 6 7 8
2015-02-07 01:10:58 Stored cookie google.pt -1 (ANY) / <permanent> <insecure> [expiry 2015-08-09 02:10:58] NID 67=AkXpY2nJPDDcH7xKJkslxdCtflnhOZJiNwZdu4YBAIc2FnjIZIAYHzFuln5boxiOHq1WWBdbcTnLXwPqOrfxOxkLXtO2U5UAVBCU0nVcgyC61_YLZLXGR0Fmdi9M_fIp
2015-02-07 01:10:58 Registered socket 3 for persistent reuse.
2015-02-07 01:10:58 URI content encoding = `ISO-8859-1'
2015-02-07 01:10:58 Length: unspecified [text/html]
2015-02-07 01:10:58 Saving to: `index.html'
2015-02-07 01:10:58
2015-02-07 01:10:58      0K .......... .......                                     17.6M=0.001s
2015-02-07 01:10:58
2015-02-07 01:10:58 2015-02-07 01:10:58 (17.6 MB/s) - `index.html' saved [18301]
2015-02-07 01:10:58
2015-02-07 01:10:58 Removing file due to --delete-after in main():
2015-02-07 01:10:58 Removing index.html.

real    0m0.350s
user    0m0.038s
sys     0m0.027s
xbian@xbian ~ $ time wget -d -v --no-check-certificate --delete-after -4 https://www.google.pt 2>&1  | awk '{ print strftime("%Y-%m-%d %H:%M:%S"), $0; fflush(); }'
2015-02-07 01:11:01 Setting --verbose (verbose) to 1
2015-02-07 01:11:01 Setting --check-certificate (checkcertificate) to 0
2015-02-07 01:11:01 Setting --delete-after (deleteafter) to 1
2015-02-07 01:11:01 Setting --inet4-only (inet4only) to 1
2015-02-07 01:11:01 DEBUG output created by Wget 1.13.4 on linux-gnueabihf.
2015-02-07 01:11:01
2015-02-07 01:11:01 URI encoding = `UTF-8'
2015-02-07 01:11:01 --2015-02-07 01:11:01--  https://www.google.pt/
2015-02-07 01:11:06 Resolving www.google.pt (www.google.pt)... 213.30.5.25, 213.30.5.53, 213.30.5.38, ...
2015-02-07 01:11:06 Caching www.google.pt => 213.30.5.25 213.30.5.53 213.30.5.38 213.30.5.32 213.30.5.24 213.30.5.46 213.30.5.39 213.30.5.18 213.30.5.52 213.30.5.31 213.30.5.59 213.30.5.45
2015-02-07 01:11:06 Connecting to www.google.pt (www.google.pt)|213.30.5.25|:443... connected.
2015-02-07 01:11:06 Created socket 4.
2015-02-07 01:11:06 Releasing 0x00b53d48 (new refcount 1).
2015-02-07 01:11:06
2015-02-07 01:11:06 ---request begin---
2015-02-07 01:11:06 GET / HTTP/1.1
2015-02-07 01:11:06 User-Agent: Wget/1.13.4 (linux-gnueabihf)
2015-02-07 01:11:06 Accept: */*
2015-02-07 01:11:06 Host: www.google.pt
2015-02-07 01:11:06 Connection: Keep-Alive
2015-02-07 01:11:06
2015-02-07 01:11:06 ---request end---
2015-02-07 01:11:06 HTTP request sent, awaiting response...
2015-02-07 01:11:06 ---response begin---
2015-02-07 01:11:06 HTTP/1.1 200 OK
2015-02-07 01:11:06 Date: Sat, 07 Feb 2015 01:11:06 GMT
2015-02-07 01:11:06 Expires: -1
2015-02-07 01:11:06 Cache-Control: private, max-age=0
2015-02-07 01:11:06 Content-Type: text/html; charset=ISO-8859-1
2015-02-07 01:11:06 Set-Cookie: PREF=ID=579b1dd2360c9122:FF=0:TM=1423271466:LM=1423271466:S=9zOSotidcZWjJfXX; expires=Mon, 06-Feb-2017 01:11:06 GMT; path=/; domain=.google.pt
2015-02-07 01:11:06 Set-Cookie: NID=67=Jetj6llJijt09db9ekqGS6cBo3DE0CDqfQkp9Sh8xtLyYnNGU5zHoMED0whNkToP_w6mk6-oLTSRVdYIDekUEZH02oBYQPQhHmhpQzENI08zGNg9Jxn4EkXTIVApLCAG; expires=Sun, 09-Aug-2015 01:11:06 GMT; path=/; domain=.google.pt; HttpOnly
2015-02-07 01:11:06 P3P: CP="This is not a P3P policy! See http://www.google.com/support/accounts/bin/answer.py?hl=en&answer=151657 for more info."
2015-02-07 01:11:06 Server: gws
2015-02-07 01:11:06 X-XSS-Protection: 1; mode=block
2015-02-07 01:11:06 X-Frame-Options: SAMEORIGIN
2015-02-07 01:11:06 Accept-Ranges: none
2015-02-07 01:11:06 Vary: Accept-Encoding
2015-02-07 01:11:06 Transfer-Encoding: chunked
2015-02-07 01:11:06
2015-02-07 01:11:06 ---response end---
2015-02-07 01:11:06 200 OK
2015-02-07 01:11:06 cdm: 1 2 3 4 5 6 7 8
2015-02-07 01:11:06 Stored cookie google.pt -1 (ANY) / <permanent> <insecure> [expiry 2017-02-06 01:11:06] PREF ID=579b1dd2360c9122:FF=0:TM=1423271466:LM=1423271466:S=9zOSotidcZWjJfXX
2015-02-07 01:11:06 cdm: 1 2 3 4 5 6 7 8
2015-02-07 01:11:06 Stored cookie google.pt -1 (ANY) / <permanent> <insecure> [expiry 2015-08-09 02:11:06] NID 67=Jetj6llJijt09db9ekqGS6cBo3DE0CDqfQkp9Sh8xtLyYnNGU5zHoMED0whNkToP_w6mk6-oLTSRVdYIDekUEZH02oBYQPQhHmhpQzENI08zGNg9Jxn4EkXTIVApLCAG
2015-02-07 01:11:06 Registered socket 4 for persistent reuse.
2015-02-07 01:11:06 URI content encoding = `ISO-8859-1'
2015-02-07 01:11:06 Length: unspecified [text/html]
2015-02-07 01:11:06 Saving to: `index.html'
2015-02-07 01:11:06
2015-02-07 01:11:06      0K .......... .......                                      670K=0.03s
2015-02-07 01:11:06
2015-02-07 01:11:06 2015-02-07 01:11:06 (670 KB/s) - `index.html' saved [18319]
2015-02-07 01:11:06
2015-02-07 01:11:06 Removing file due to --delete-after in main():
2015-02-07 01:11:06 Removing index.html.

real    0m5.371s
user    0m4.083s
sys     0m0.280s

在卷曲中，差别似乎没有那么大.

xbian@xbian ~ $ curl -V
curl 7.26.0 (arm-unknown-linux-gnueabihf) libcurl/7.26.0 OpenSSL/1.0.1e zlib/1.2.7 libidn/1.25 libssh2/1.4.2 librtmp/2.3
Protocols: dict file ftp ftps gopher http https imap imaps ldap pop3 pop3s rtmp rtsp scp sftp smtp smtps telnet tftp
Features: Debug GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz TLS-SRP
xbian@xbian ~ $ time curl -s http:///www.google.pt > /dev/null

real    0m0.140s
user    0m0.056s
sys     0m0.034s
xbian@xbian ~ $ time curl -s https:///www.google.pt > /dev/null

real    0m0.294s
user    0m0.060s
sys     0m0.031s

Answer 1

在与wget开发人员进行了一些测试和讨论之后，我得出的结论是，这是由于gnutls库造成的。如果使用openssl编译wget，则行为更像是curl。

Answer 2

设置SSL/TSL存在一些开销，因为必须建立会话密钥，但是这往往是可以忽略不计的，因此我怀疑这是否是真正的原因，但我们永远不知道。

Answer 3

我怎样才能克服这一切？

你不能这么做。

HTTP和HTTPS的区别在于后者使用SSL/TLS来保护连接。SSL/TLS有大量的间接费用：

• 启动时，客户端和服务器交换证书等，以便(至少)客户端可以验证服务器不是冒牌货。 启动协商需要许多客户端<->服务器消息交换。如果TCP/IP级连接有明显的延迟，这将表现为明显的延迟。
• 连接建立后，经过连接的数据在发送时加密，在接收时解密。

我认为HTTPS没有任何实用的替代方案--您想要安全地与常规的、当前一代的web服务器进行对话。我认为它不会随着“下一代”HTTP而改变，即HTTP/2。

要加快速度(HTTP/1.1或HTTP/2)，唯一可以做的事情是为多个GET重用一个“持久连接”。SSL/TLS协商仅在建立连接时发生。但是，持久连接在“单击”用例中没有帮助；例如，当您使用wget或curl来获取一个文件时。