Capistrano失败的原因是:证书验证失败(无法获取本地颁发者证书)

Question

实际上，我们有两个集群，每个集群都有两个CentOS 7.5服务器。系统1是具有通配符证书的开发集群；系统2是在前端服务器上具有非通配符证书但在后端服务器上具有通配符证书的生产集群。

我们在Apache上运行Ruby on Rails和Passenger，并与Capistrano一起部署。我们正在尝试使用亚马逊网络服务密钥管理服务(KMS)部署对称加密gem来存储客户主密钥(https://rocketjob.github.io/symmetric-encryption/configuration.html)。

经过一些工作，System 1(开发)中的两台服务器都部署并成功运行。我们能够成功地部署到System 2(生产)的后端服务器，但是前端--使用非通配符证书的服务器--在Capistrano部署的deploy:assets:precompile阶段失败了：

01 Seahorse::Client::NetworkingError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)
      01 /var/www/web-2/html/shared/bundle/ruby/2.6.0/gems/aws-sdk-core-3.89.1/lib/seahorse/client/net_http/connection_pool.rb:299:in `start_session'
...
01 /var/www/web-2/html/shared/bundle/ruby/2.6.0/gems/aws-sdk-kms-1.28.0/lib/aws-sdk-kms/client.rb:1375:in `decrypt'
      01 /var/www/web-2/html/shared/bundle/ruby/2.6.0/gems/symmetric-encryption-4.3.1/lib/symmetric_encryption/utils/aws.rb:56:in `block in decrypt'
      01 /var/www/web-2/html/shared/bundle/ruby/2.6.0/gems/symmetric-encryption-4.3.1/lib/symmetric_encryption/utils/aws.rb:129:in `auto_create_master_key'
      01 /var/www/web-2/html/shared/bundle/ruby/2.6.0/gems/symmetric-encryption-4.3.1/lib/symmetric_encryption/utils/aws.rb:55:in `decrypt'
      01 /var/www/web-2/html/shared/bundle/ruby/2.6.0/gems/symmetric-encryption-4.3.1/lib/symmetric_encryption/keystore/aws.rb:137:in `read'
      01 /var/www/web-2/html/shared/bundle/ruby/2.6.0/gems/symmetric-encryption-4.3.1/lib/symmetric_encryption/keystore.rb:168:in `read_key'
      01 /var/www/web-2/html/shared/bundle/ruby/2.6.0/gems/symmetric-encryption-4.3.1/lib/symmetric_encryption/cipher.rb:22:in `from_config'
      01 /var/www/web-2/html/shared/bundle/ruby/2.6.0/gems/symmetric-encryption-4.3.1/lib/symmetric_encryption/config.rb:87:in `block in ciphers'

仅供参考，在每台生产服务器上运行openssl s_client -connect s3-us-west-2.amazonaws.com:443会产生不同的结果，前端服务器会抛出错误。

下面是带有通配符证书的后端服务器：

-bash-4.2$  openssl s_client -connect s3-us-west-2.amazonaws.com:443
CONNECTED(00000003)
depth=2 C = IE, O = Baltimore, OU = CyberTrust, CN = Baltimore CyberTrust Root
verify return:1
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Baltimore CA-2 G2
verify return:1
depth=0 C = US, ST = Washington, L = Seattle, O = "Amazon.com, Inc.", CN = *.s3-us-west-2.amazonaws.com
verify return:1
---
Certificate chain
 0 s:/C=US/ST=Washington/L=Seattle/O=Amazon.com, Inc./CN=*.s3-us-west-2.amazonaws.com
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Baltimore CA-2 G2
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Baltimore CA-2 G2
   i:/C=IE/O=Baltimore/OU=CyberTrust/CN=Baltimore CyberTrust Root
---
Server certificate
-----BEGIN CERTIFICATE-----
     --%<--snip-->%--
-----END CERTIFICATE-----
subject=/C=US/ST=Washington/L=Seattle/O=Amazon.com, Inc./CN=*.s3-us-west-2.amazonaws.com
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Baltimore CA-2 G2
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3727 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: B001EA6EC9FC5049EEE85A13DA0373634992E8AB907E4558BE64F5A26055223D
    Session-ID-ctx: 
    Master-Key: DE8B5EBE0645974504E83FB6AE73CB54042EDA6B13FCC32A0B6C601EC2231E3627FE721ECE1F07CA48915D2A69195C67
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1581341714
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
closed

下面是带有非通配符证书的后端服务器：

-bash-4.2$  openssl s_client -connect s3-us-west-2.amazonaws.com:443
CONNECTED(00000003)
depth=1 C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert Baltimore CA-2 G2
verify error:num=20:unable to get local issuer certificate
---
Certificate chain
 0 s:/C=US/ST=Washington/L=Seattle/O=Amazon.com, Inc./CN=*.s3-us-west-2.amazonaws.com
   i:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Baltimore CA-2 G2
 1 s:/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Baltimore CA-2 G2
   i:/C=IE/O=Baltimore/OU=CyberTrust/CN=Baltimore CyberTrust Root
---
Server certificate
-----BEGIN CERTIFICATE-----
     --%<--snip-->%--
-----END CERTIFICATE-----
subject=/C=US/ST=Washington/L=Seattle/O=Amazon.com, Inc./CN=*.s3-us-west-2.amazonaws.com
issuer=/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert Baltimore CA-2 G2
---
No client certificate CA names sent
Peer signing digest: SHA256
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3727 bytes and written 415 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 002173BC9539FF9A0CAEAF4EE3699102D638784A30F505AAE679F26E0DA22EA0
    Session-ID-ctx: 
    Master-Key: 252EF43EC4EB107FA27702CA5EDBEE894F8BAB9FD7B326EB581F49A666F21988543B7F0EF185F2E7D4D13E7E052A8B98
    Key-Arg   : None
    Krb5 Principal: None
    PSK identity: None
    PSK identity hint: None
    Start Time: 1581341716
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---
closed

我们认为我们的证书链可能有问题，但在阅读并遵循https://medium.com/@superseb/get-your-certificate-chain-right-4b117a9c0fce中的说明后，我们确认我们的证书和证书链是正常的：

-bash-4.2$ openssl verify company.net.crt
company.net.crt: OK

-bash-4.2$ openssl x509 -in company.net.crt -noout -issuer
issuer= /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2

-bash-4.2$ openssl x509 -noout -subject -in ca-bundle.crt
subject= /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2

-bash-4.2$ openssl verify -CAfile ca-bundle.crt company.net.crt
company.net.crt: OK

因此，根据上下文，我们可以从openssl获得两种不同的结果。

在一段时间内，我们还尝试按照https://github.com/rubygems/rubygems/issues/2415#issuecomment-460998632使用Aws.use_bundled_cert!初始化亚马逊网络服务，但这没有帮助。我已经看到了执行以下命令的参考，但我们还没有尝试，因为我们想要更深入地研究这一点(看看这是否是这个网站上的人的共识)：

curl -fsSL curl.haxx.se/ca/cacert.pem -o "$(ruby -ropenssl -e 'puts OpenSSL::X509::DEFAULT_CERT_FILE')"

因此，假设我们只有一个不使用通配符证书的系统有问题，似乎我们的服务器、对称加密gem或AWS ( KMS或CMS)之间的某处存在配置错误。

显然，我们很乐意为我们的问题找到一个解决方案，但即使是关于如何诊断它的建议也会非常受欢迎。请注意，这是我们第一次使用AWS KMS (除了对称加密gem之外)。

Answer 1

在这种情况下，/etc/pki/tls/certs中的文件ca-bundle.crt似乎已经被践踏了。实际上，这个文件是一个指向/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem的符号链接，它实际上就是被删除的那个文件。幸运的是，由于我们有几个相同的系统，我们能够复制一个未损坏的tls-ca-bundle.pem，openssl测试工作正常，Capistrano可以完成其部署。因此，这似乎更多的是系统管理问题，而不是软件开发问题。我只是碰巧用AWS KMS添加对称加密gem暴露了这个问题。