使用tdagent将aws-aws与fluentd连接起来

Question

我已使用tdagent成功连接了本地计算机上的elasticsearch，但在过渡环境中，我需要连接aws elasticsearch，如果我使用ruby安装了fluentd，则会有一个插件

gem 'fluent-plugin-aws-elasticsearch-service'

但是，当使用fluentd和tdagent时，我们如何配置aws-elasticsearch？

<match catalog>
  @type elasticsearch
  host https://elastic_devel_aws.com
  # port 9200
  logstash_format true 
  include_timestamp true 
  index_name _logs_test
  flush_interval 10s
  reconnect_on_error true
  reload_on_failure true
  reload_connections false
  request_timeout 120s
  <buffer>
    @type file
   flush_interval 10s
   retry_type periodic
   retry_forever true
   retry_wait 10s
   chunk_limit_size 16Mb
   queue_limit_length 4096
   total_limit_size 60Gb
   path /var/lib/td-agent/buffers/output_elasticsearch-1
 </buffer>
</match> 

当尝试使用这样的东西时，

  Could not communicate to Elasticsearch, resetting connection and trying again. getaddrinfo: Name or service not known (SocketError)

Answer 1

要使用td-agent安装gem，需要执行以下命令

/usr/sbin/td-agent-gem install fluent-plugin-aws-elasticsearch-service'

您可以使用以下配置将日志发送到AWS托管elastcisearch。

<match *.**>
  @type elasticsearch

  host "#{ENV['ELASTICSEARCH_HOST']}"
  port "#{ENV['ELASTICSEARCH_PORT']}"
  scheme "#{ENV['ELASTICSEARCH_SCHEME'] || 'http'}"
  type_name log

  include_tag_key true
  tag_key @logtag

  logstash_format true
  logstash_prefix fos
  logstash_dateformat %Y%m%d

  reconnect_on_error true

  <buffer>
    @type file
    path /fluentd/log/fos/elastic-buffer
    flush_thread_count 8
    flush_interval 1s
    chunk_limit_size 32M
    queue_limit_length 4
    flush_mode interval
    retry_max_interval 30
    retry_forever true
  </buffer>
</match>

您还需要确保允许staging环境到达弹性搜索实例，您可以通过执行以下命令进行检查

telnet elastic_devel_aws.com 443