如何使用FluentD和ELK ElasticStack配置证书授权

Question

我在将FluentD安装在Amazon集群中时遇到问题，该集群将直接将数据发送到Azure中的ElasticSearch堆栈。我想像您使用一个证书(ca.pem、cert.pem和cert.key)来配置File节拍一样来配置它，而不是使用用户/密码身份验证。

我已经成功地启动和运行了FluentD文件，并且RBAC按其应有的方式工作，并且由于证书授权的文档似乎不存在，所以我尝试了一些尝试和错误，但是没有vail。

我对证书的配置如下：

apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: elasticsearch-azure
---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd-role
  namespace: elastisearch-azure
rules:
  - apiGroups: [""]
    resources:
      - namespaces
      - pods
      - pods/logs
    verbs: ["get", "list", "watch"]
---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: fluentd-role-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: fluentd-role
subjects:
  - kind: ServiceAccount
    name: fluentd
    namespace: elasticsearch-azure
---

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: elasticsearch-azure
  labels:
    k8s-app: fluentd-logging
    version: v1
spec:
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      serviceAccountName: fluentd
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch
        envFrom:
        - secretRef:
            name: fluent-tls
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: "{{server_namne}}"
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: "{port}"
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: "https"
          # Option to configure elasticsearch plugin with self signed certs
          # ================================================================
          - name: FLUENT_ELASTICSEARCH_SSL_VERIFY
            value: "true"
          # Option to configure elasticsearch plugin with tls
          # ================================================================
          - name: FLUENT_ELASTICSEARCH_SSL_VERSION
            value: "TLSv1_2"
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: ssl
          mountPath: /fluent-tls/ssl
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      # certificates folder for filebeat
      - name: ssl
        secret:
          secretName: fluent-tls

我使用以下命令创建了这个秘密：

kubectl create secret generic fluent-tls \
--from-file=ca_file=./chain.pem \
--from-file=cert_pem=./cert.pem \
--from-file=cert_key=./cert.key

我在运行豆荚时遇到的错误如下所示：

 <match **>
    @type elasticsearch
    @id out_es
    @log_level "info"
    include_tag_key true
    host "super-sercret-host.com"
    port even-more-secret-portnumber
    path ""
    scheme https
    ssl_verify false
    ssl_version TLSv1_2
    reload_connections false
    reconnect_on_error true
    reload_on_failure true
    log_es_400_reason false
    logstash_prefix "logstash"
    logstash_format true
    index_name "logstash"
    type_name "fluentd"
    <buffer>
      flush_thread_count 8
      flush_interval 5s
      chunk_limit_size 2M
      queue_limit_length 32
      retry_max_interval 30
      retry_forever true
    </buffer>
  </match>
</ROOT>
2019-12-16 14:51:30 +0000 [info]: starting fluentd-1.7.4 pid=6 ruby="2.6.5"
2019-12-16 14:51:30 +0000 [info]: spawn command to main:  cmdline=["/usr/local/bin/ruby", "-Eascii-8bit:ascii-8bit", "/fluentd/vendor/bundle/ruby/2.6.0/bin/fluentd", "-c", "/fluentd/etc/fluent.conf", "-p", "/fluentd/plugins", "--gemfile", "/fluentd/Gemfile", "--under-supervisor"]
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-concat' version '2.4.0'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.13'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '3.7.1'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-grok-parser' version '2.6.1'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-json-in-json-2' version '1.0.2'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '2.3.0'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-multi-format-parser' version '1.0.0'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-prometheus' version '1.6.1'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.0.1'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.2.0'
2019-12-16 14:51:31 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.2'
2019-12-16 14:51:31 +0000 [info]: gem 'fluentd' version '1.7.4'
2019-12-16 14:51:31 +0000 [info]: adding match pattern="fluent.**" type="null"
2019-12-16 14:51:31 +0000 [info]: adding filter pattern="kubernetes.**" type="kubernetes_metadata"
2019-12-16 14:51:31 +0000 [info]: adding match pattern="**" type="elasticsearch"
2019-12-16 14:51:34 +0000 [warn]: #0 [out_es] Could not communicate to Elasticsearch, resetting connection and trying again. SSL_connect returned=1 errno=0 state=error: sslv3 alert handshake failure (OpenSSL::SSL::SSLError)
2019-12-16 14:51:34 +0000 [warn]: #0 [out_es] Remaining retry: 14. Retry to communicate after 2 second(s).
2019-12-16 14:51:38 +0000 [warn]: #0 [out_es] Could not communicate to Elasticsearch, resetting connection and trying again. SSL_connect returned=1 errno=0 state=error: sslv3 alert handshake failure (OpenSSL::SSL::SSLError)
2019-12-16 14:51:38 +0000 [warn]: #0 [out_es] Remaining retry: 13. Retry to communicate after 4 second(s).

我知道，如果我只能将配置属性: ca_file、client_pem和client_key参数放到标记中，它可能会工作，但到目前为止我还没有做到这一点。任何帮助都是非常感谢的。

Answer 1

谢谢!非常感谢您；您的配置示例帮助解决了我在FLuentD和ES之间的SSL问题，现在也许我可以帮助您。

在Bitnami的FluentD中，我正在使用Open进行Elasticsearch搜索，而且我也收到了类似的错误。我的决心是使用你的配置的一部分，但我必须改变主机主机，像这样；

hosts               https://admin:admin@odfe-node1:9200

似乎您必须在一行上指定用户名、协议和端口，就像必须在集群中指定多个主机一样。对我起作用了。查看FluentD文档以获得进一步的参考。