Kafka Producer中SSL握手失败

Question

我正在尝试设置一个启用SSL通信的Kafka broker。我遵循了7.2下的Kafka文档中描述的指南：https://kafka.apache.org/documentation/#security

当我尝试执行命令./kafka-console-producer.sh --broker-list ec2-ip.eu-central-1.compute.amazonaws.com:9093 -topic test --producer.config ../config/producer.properties时，我得到：

ERROR [Producer clientId=console-producer] Connection to node -1 (ec2-ip.eu-central-1.compute.amazonaws.com/172.31.42.33:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)
[2019-08-22 13:15:56,728] ERROR [Producer clientId=console-producer] Connection to node -1 (ec2-ip.eu-central-1.compute.amazonaws.com/172.31.42.33:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)

测试SSl连接的一般方法

openssl s_client -debug -connect ec2-ip.eu-central-1.compute.amazonaws.com:9093 -tls1

结果：

0020 - a0 41 45 81 42 b6 83 d3-2b 94 02 9f ac 42 73 42   .AE.B...+....BsB
---
Certificate chain
 0 s:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=Unknown
   i:/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=Unknown
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDuTCCAqGgAwIBAgIEPsshdTANBgkqhkiG9w0BAQsFADBsMRAwDgYDVQQGEwdV
bmtub3duMRAwDgYDVQQIEwdVbmtub3duMRAwDgYDVQQHEwdVbmtub3duMRAwDgYD
VQQKEwdVbmtub3duMRAwDgYDVQQLEwdVbmtub3duMRAwDgYDVQQDEwdVbmtub3du
MB4XDTE5MDgyMjEyNTIwOFoXDTIwMDgyMTEyNTIwOFowbDEQMA4GA1UEBhMHVW5r
bm93bjEQMA4GA1UECBMHVW5rbm93bjEQMA4GA1UEBxMHVW5rbm93bjEQMA4GA1UE
ChMHVW5rbm93bjEQMA4GA1UECxMHVW5rbm93bjEQMA4GA1UEAxMHVW5rbm93bjCC
ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAI5EKxmSDLQvW/ePu9mB3qdf
E4cG1ok1kI48S6yL4gchx2egIC6Zjr5gBhuek36mhKV82H1nVYfDzJVyxoyaTZ5q
ux8xq+ooNN/chKHy0ExpCZ2HHvrQiTPQit//dyA5zc5yOyRcJo/m2EhYfRMuLfFM
kx1mEg2cs6Qa/hd6jKjCSvbcKDfrOhf4rLagHzI+2GRQfF6LXxYBE3Ti7u6fkqog
o67CTrNR6IER1b4wGIeHga8eXiVhOuP4JiWdpF69i5py0RVfzsesP0W8RB2qElte
+7EwShaFO72Xa7MPiDz1rE4D5pohQ09AWGb/B0YIcI7qh9jyInbj4eWhhlnd1lsC
AwEAAaNjMGEwQAYDVR0RBDkwN4I1ZWMyLTM1LTE1OC0yMzItMTc2LmV1LWNlbnRy
YWwtMS5jb21wdXRlLmFtYXpvbmF3cy5jb20wHQYDVR0OBBYEFNGugrNUmFiVKVyU
Uiepd0OUme1wMA0GCSqGSIb3DQEBCwUAA4IBAQAToYYUsvQ6MlOQ+HHWnfax5L0u
M/IrExjcrSmnxAevnTem5+Hnz+AmlRHSF1/FfCWhi2XMSHBHXU52uhIt+kDLNhFh
D17Je0FydthW8roBsLZ+XsuHjBi8MXo41YAA19+cjW60/se3+kWxOHH1rEfV9ky/
K9kWufQPl3U3fbUzh0rWwriSDaPUTjHg/IeROhVe8gt/QeBfTU2KUqnln6K58PfC
kE2c2zCTQwpsfwi5gqvrdqWZc/A5qriv1l39zKU1icfTZzdeejcjtbCdFqajOqar
lMXwyl1bPuPiOHas7K9J9MSREQYql89gbN5V9gO//IHmM97eLt4npFSpu8Cp
-----END CERTIFICATE-----
subject=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=Unknown
issuer=/C=Unknown/ST=Unknown/L=Unknown/O=Unknown/OU=Unknown/CN=Unknown
---
No client certificate CA names sent
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1451 bytes and written 236 bytes
Verification error: self signed certificate
---
New, TLSv1.0, Cipher is ECDHE-RSA-AES256-SHA
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1
    Cipher    : ECDHE-RSA-AES256-SHA
    Session-ID: 5D5E9882C4E8DD80427108B85C7649F0EC1AA87B9DAA0EA1FCE8C7063C83A61B
    Session-ID-ctx: 
    Master-Key: 861677EDA9E19E3D8926889A9B0DE299593C7FCD49DB8A55EBF4D222800169E16CDB74DCE0EC392A3B491268FCCF5F07
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1566480514
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
    Extended master secret: yes
---

在我看来没问题。

代理的配置：

listeners=SSL://ec2-ip.eu-central-1.compute.amazonaws.com:9093,PLAINTEXT://ec2-ip.eu-central-1.compute.amazonaws.com:9092
group.initial.rebalance.delay.ms=0
advertised.listeners=PLAINTEXT://ec2-ip.eu-central-1.compute.amazonaws.com:9092,SSL://ec2-ip.eu-central-1.compute.amazonaws.com:9093
ssl.endpoint.identification.algorithm=SSL

ssl.keystore.location=/kafka/kafka_2.12-2.2.0/config/certs/server.keystore.jks
ssl.keystore.password=Welcome01
ssl.key.password=Welcome01
ssl.truststore.location=/kafka/kafka_2.12-2.2.0/config/certs/server.truststore.jks
ssl.truststore.password=Welcome01
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.keystore.type=JKS
ssl.truststore.type=JKS

生产者的配置：

bootstrap.servers=localhost:9092
security.protocol=SSL
ssl.truststore.location=/kafka/kafka_2.12-2.2.0/config/certs/client.truststore.jks
ssl.truststore.password=Welcome01
logging.level.org.apache.kafka: DEBUG
log4j.rootLogger=DEBUG

您是否看到一些配置错误？我怎样才能继续深入研究这个问题呢？

多亏了mazaneiche，我才能得到导致握手失败的错误：

kafka-producer-network-thread | console-producer, fatal error: 46: General SSLEngine problem
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
%% Invalidated:  [Session-3, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384]
kafka-producer-network-thread | console-producer, SEND TLSv1.2 ALERT:  fatal, description = certificate_unknown

为什么会出现这个错误？如何修复呢？

Answer 1

显然这里不是问题所在，但我已经为同样的错误挣扎了一段时间，在我的例子中，kafka-console-producer.sh失败了，并出现了类似的错误：

$ kafka-console-producer.sh --broker-list 192.168.1.173:9093 --topic kafka-security-topic --producer.config ~/ssl/client-ssl-auth.properties
org.apache.kafka.common.errors.SslAuthenticationException: SSL handshake failed
Caused by: javax.net.ssl.SSLHandshakeException: General SSLEngine problem

使用debug检查

$ export KAFKA_OPTS="-Djavax.net.debug=ssl"

kafka-producer-network-thread | console-producer, fatal error: 46: General SSLEngine problem
java.security.cert.CertificateException: No subject alternative names present
%% Invalidated:  [Session-1, TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384]
kafka-producer-network-thread | console-producer, SEND TLSv1.2 ALERT:  fatal, description = certificate_unknown
[2021-07-13 16:21:53,909] ERROR [Producer clientId=console-producer] Connection to node -1 (/192.168.1.173:9093) failed authentication due to: SSL handshake failed (org.apache.kafka.clients.NetworkClient)

在我的例子中，原因是在指定broker-list参数时使用IP而不是主机名。

这是可行的：

$ kafka-console-producer.sh --broker-list red03:9093 --topic kafka-security-topic --producer.config ~/ssl/client-ssl-auth.properties