istio-代理403错误：“上游连接错误或头前断开/重置。重置原因:连接失败”

Question

我们在istio入口网关后面部署了一个应用程序，可以在test.domain.com/jenkinscore.We上访问，我们使用了istio1.4.5。域名是为istio入口网关服务IP创建的。按照下面的日志，当我们点击这个URL时，istio将抛出一个403错误：upstream connect error or disconnect/reset before headers. reset reason: connection failure。下面是原木。这只是间歇性的。在重新启动入口网关吊舱时，问题得到解决。有人能告诉我们这一错误的原因是什么吗？

:42:20.798][46][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:259] [C2469] new stream
[Envoy (Epoch 1)] [2020-06-09 11:42:20.798][46][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:483] [C2469] recv frame type=1
[Envoy (Epoch 1)] [2020-06-09 11:42:20.798][46][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:708] [C2469][S10386582713969444678] request headers complete (end_stream=true):
':method', 'GET'
':authority', 'test.domain.com'
':scheme', 'https'
':path', '/jenkinscore'
'cache-control', 'max-age=0'
'upgrade-insecure-requests', '1'
'user-agent', 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36'
'accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9'
'sec-fetch-site', 'cross-site'
'sec-fetch-mode', 'navigate'
'sec-fetch-user', '?1'
'sec-fetch-dest', 'document'
'accept-encoding', 'gzip, deflate, br'
'accept-language', 'en-US,en;q=0.9'

[Envoy (Epoch 1)] [2020-06-09 11:42:20.798][46][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1257] [C2469][S10386582713969444678] request end stream
[Envoy (Epoch 1)] [2020-06-09 11:42:20.798][46][debug][jwt] [external/envoy/source/extensions/filters/http/jwt_authn/filter.cc:101] Called Filter : setDecoderFilterCallbacks
[Envoy (Epoch 1)] [2020-06-09 11:42:20.798][46][debug][filter] [src/envoy/http/mixer/filter.cc:47] Called Mixer::Filter : Filter
[Envoy (Epoch 1)] [2020-06-09 11:42:20.798][46][debug][filter] [src/envoy/http/mixer/filter.cc:148] Called Mixer::Filter : setDecoderFilterCallbacks
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][filter] [external/envoy/source/extensions/filters/http/ext_authz/ext_authz.cc:80] [C2469][S10386582713969444678] ext_authz filter calling authorization server
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][debug][router] [external/envoy/source/common/router/router.cc:434] [C0][S9059969016458298666] cluster 'ext_authz' match for URL '/envoy.service.auth.v2.Authorization/Check'
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][debug][router] [external/envoy/source/common/router/router.cc:549] [C0][S9059969016458298666] router decoding headers:
':method', 'POST'
':path', '/envoy.service.auth.v2.Authorization/Check'
':authority', 'ext_authz'
':scheme', 'http'
'te', 'trailers'
'grpc-timeout', '10000m'
'content-type', 'application/grpc'
'x-b3-traceid', 'a4xxxx3471f0f7496063d056b2d9'
'x-b3-spanid', '7a236se1c6c190'
'x-b3-parentspanid', 'f7496063d056b2d9'
'x-b3-sampled', '0'
'x-envoy-internal', 'true'
'x-forwarded-for', '10.48.3.5'
'x-envoy-expected-rq-timeout-ms', '10000'

[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][debug][client] [external/envoy/source/common/http/codec_client.cc:31] [C2470] connecting
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][debug][connection] [external/envoy/source/common/network/connection_impl.cc:711] [C2470] connecting to 127.0.0.1:10003
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][debug][connection] [external/envoy/source/common/network/connection_impl.cc:720] [C2470] connection in progress
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][debug][http2] [external/envoy/source/common/http/http2/codec_impl.cc:912] [C2470] setting stream-level initial window size to 268435456
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][debug][http2] [external/envoy/source/common/http/http2/codec_impl.cc:934] [C2470] updating connection-level initial window size to 268435456
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][debug][pool] [external/envoy/source/common/http/conn_pool_base.cc:20] queueing request due to no available connections
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][router] [external/envoy/source/common/router/router.cc:1475] [C0][S9059969016458298666] buffering 1023 bytes
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:966] [C2469][S10386582713969444678] decode headers called: filter=0x559dc3768780 status=4
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:424] [C2469] dispatched 441 bytes
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:666] [C2469] about to send frame type=4, flags=0
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:720] [C2469] send data: bytes=15
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][connection] [external/envoy/source/common/network/connection_impl.cc:398] [C2469] writing 15 bytes, end_stream false
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:608] [C2469] sent frame type=4
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:666] [C2469] about to send frame type=4, flags=1
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:720] [C2469] send data: bytes=9
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][connection] [external/envoy/source/common/network/connection_impl.cc:398] [C2469] writing 9 bytes, end_stream false
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:608] [C2469] sent frame type=4
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:666] [C2469] about to send frame type=8, flags=0
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:720] [C2469] send data: bytes=13
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][connection] [external/envoy/source/common/network/connection_impl.cc:398] [C2469] writing 13 bytes, end_stream false
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][http2] [external/envoy/source/common/http/http2/codec_impl.cc:608] [C2469] sent frame type=8
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][connection] [external/envoy/source/common/network/connection_impl.cc:462] [C2469] socket event: 2
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][connection] [external/envoy/source/common/network/connection_impl.cc:550] [C2469] write ready
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:259] [C2469] ssl write returns: 37
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][connection] [external/envoy/source/common/network/connection_impl.cc:462] [C2470] socket event: 3
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][connection] [external/envoy/source/common/network/connection_impl.cc:550] [C2470] write ready
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][debug][connection] [external/envoy/source/common/network/connection_impl.cc:568] [C2470] delayed connection error: 111
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][debug][connection] [external/envoy/source/common/network/connection_impl.cc:193] [C2470] closing socket: 0
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][debug][client] [external/envoy/source/common/http/codec_client.cc:88] [C2470] disconnect. resetting 0 pending requests
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][debug][pool] [external/envoy/source/common/http/http2/conn_pool.cc:152] [C2470] client disconnected
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][debug][router] [external/envoy/source/common/router/router.cc:911] [C0][S9059969016458298666] upstream reset: reset reason connection failure
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][debug][http] [external/envoy/source/common/http/async_client_impl.cc:93] async http request response headers (end_stream=true):
':status', '200'
'content-type', 'application/grpc'
'grpc-status', '14'
'grpc-message', 'upstream connect error or disconnect/reset before headers. reset reason: connection failure'

[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][filter] [external/envoy/source/extensions/filters/http/ext_authz/ext_authz.cc:244] [C2469][S10386582713969444678] ext_authz filter rejected the request with an error. Response status code: 403
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1354] [C2469][S10386582713969444678] Sending local reply with details ext_authz_error
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1441] [C2469][S10386582713969444678] encode headers called: filter=0x559dc3646d20 status=0
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1441] [C2469][S10386582713969444678] encode headers called: filter=0x559dc3554730 status=0
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][debug][filter] [src/envoy/http/mixer/filter.cc:135] Called Mixer::Filter : encodeHeaders 0
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][trace][http] [external/envoy/source/common/http/conn_manager_impl.cc:1441] [C2469][S10386582713969444678] encode headers called: filter=0x559dc35ce1e0 status=0
[Envoy (Epoch 1)] [2020-06-09 11:42:20.799][46][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1552] [C2469][S10386582713969444678] encoding headers via codec (end_stream=true):
':status', '403'
'date', 'Tue, 09 Jun 2020 11:42:20 GMT'
'server', 'istio-envoy'

Answer 1

正如我在评论中提到的

公使文件说那

status_on_error设置当过滤器和授权服务器之间存在网络错误时返回给客户端的HTTP状态。默认状态是HTTP 403禁忌。

如前所述，这里

外部授权筛选器调用授权服务以检查传入请求是否已被授权。过滤器可以配置为网络过滤器，也可以配置为HTTP过滤器，或者两者兼而有之。如果网络筛选器认为请求未经授权，则连接将被关闭。如果请求在HTTP过滤器上被认为是未经授权的，则请求将被拒绝，并有403 (禁止的)响应。

最近，我发现教程可能很有帮助，因为它们为istio设置了外部授权服务器。

在此设置中，ingresss网关将首先将入站请求头发送到另一个istio服务，该服务检查远程用户/客户端提交的标头值。如果标头值通过了某些条件，外部授权服务器将指示授权服务器继续上行请求。

我建议查看本教程，并检查istio和外部授权中的所有内容是否配置正确。

提示->建议将这些过滤器配置为过滤器链中的第一个过滤器，以便在处理请求的其他过滤器之前授权请求。外部授权服务集群可以是静态配置的，也可以是通过群集发现服务配置的。如果请求传入时外部服务不可用，则通过适用的网络过滤器或HTTP中failure_mode_allow配置的配置设置来定义请求是否被授权。如果将其设置为true，则请求将被允许(失败打开)，否则将被拒绝。默认设置为false。

我想到的另一件事可能是失效模式，如果grpc服务得到upstream connect error or disconnect/reset before headers. reset reason: connection failure，在istio中它是S 503错误。如果你看看下面的第2点，ext-authz will reject client requests and return a Forbidden response if the communication the authorization service has returned a HTTP 5xx error。所以也许值得试着把它变成真的？使用公使滤波器应该是可能的。

failure_mode_allow (bool)更改过滤器对错误的行为：

1. 当设置为true时，即使与授权服务的通信失败，或者授权服务返回了HTTP5xx错误，筛选器也将接受客户端请求。

1. 当设置为false时，如果与授权服务的通信失败，或者授权服务返回了HTTP 5xx错误，ext将拒绝客户端请求并返回禁止的响应。

希望这能有所帮助。