td-agent在更新后无法工作

Question

在Amazon Linux上进行yum更新后，Td-agent无法工作。

Td-agent 1.1.20-0起作用，但1.1.21-0不起作用，s3中没有日志数据。

错误消息是

2015-01-15 06:36:40 +0900 [error]: failed to configure sub output s3: Other '' plugin already use same buffer_path: type = , buffer_path = /var/log/td-agent/buffer/s3
2015-01-15 06:36:40 +0900 [error]: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/buf_file.rb:94:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/output.rb:188:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/output.rb:471:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-mixin-config-placeholders-0.3.0/lib/fluent/mixin/config_placeholders.rb:105:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-s3-0.4.1/lib/fluent/plugin/out_s3.rb:47:in `configure'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.2.4/lib/fluent/plugin/out_forest.rb:132:in `block in plant'
<internal:prelude>:10:in `synchronize'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.2.4/lib/fluent/plugin/out_forest.rb:128:in `plant'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluent-plugin-forest-0.2.4/lib/fluent/plugin/out_forest.rb:168:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/match.rb:36:in `emit'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/engine.rb:160:in `emit_stream'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:227:in `receive_lines'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:318:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:318:in `wrap_receive_lines'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:511:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:511:in `on_notify'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:343:in `on_notify'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:444:in `call'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:444:in `on_change'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/loop.rb:96:in `run_once'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/cool.io-1.1.1/lib/cool.io/loop.rb:96:in `run'
/usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.55/lib/fluent/plugin/in_tail.rb:212:in `run'
2015-01-15 06:36:40 +0900 [error]: Cannot output messages with tag 'apache.access'

td-agent.conf(整个配置)如下：

####
## Output descriptions:
##


## match tag=debug.** and dump to console
<match debug.**>
  type stdout
</match>

#httpd logs
<match apache.access>
  type forest
  subtype s3
  <template>
    aws_key_id MY_AWS_KEY
    aws_sec_key MY_AWS_SECRET_KEY
    s3_endpoint s3.amazonaws.com
    s3_bucket my_s3_bucket_name
    path httpd/logs/
    buffer_path /var/log/td-agent/buffer/s3
    time_slice_format %Y-%m-%d/${hostname}-%H
    time_slice_wait 10m
    buffer_chunk_limit 256m
  </template>
</match>

#not matched
<match app.**>
  type stdout
</match>

####
## Source descriptions:
##

## built-in TCP input
## @see http://docs.fluentd.org/articles/in_forward
<source>
  type forward
</source>

## built-in UNIX socket input
#<source>
#  type unix
#</source>

# HTTP input
# POST http://localhost:8888/<tag>?json=<json>
# POST http://localhost:8888/td.myapp.login?json={"user"%3A"me"}
# @see http://docs.fluentd.org/articles/in_http
#<source>
#  type http
#  port 8888
#</source>

## live debugging agent
<source>
  type debug_agent
  bind 127.0.0.1
  port 24230
</source>

## File input
## read apache logs continuously and tags td.apache.access
<source>
  type tail
  #format apache
  format ltsv
  time_format [%d/%b/%Y:%H:%M:%S %z]
  time_key time
  pos_file /var/log/td-agent/apache.access.pos
  path /path/to/apache_logfile.log
  tag apache.access
</source>

请告诉我怎么修

Answer 1

fluent-plugin-forest版本是无关的。这是时间和数量的标签问题。

Fluentd在配置阶段检查buffer_path重复。但是fluent- plugin -forest会在运行时创建插件，所以这个奇怪的复制错误会在运行时发生。

请将${tag}或其他占位符设置为buffer_path。

http://docs.fluentd.org/articles/buf_file

此外，fluent-plugin-forest现在有以下限制。所以我建议在你的配置中设置flush_at_shutdown true。

https://github.com/tagomoris/fluent-plugin-forest/issues/15

Answer 2

不幸的是，亚马逊Linux没有得到td-agent的官方支持。原因是Amazon Linux不断更改ABI (应用程序二进制接口)，并且与CentOS/REHL不兼容。

您可以使用此脚本在Amazon Linux上创建包和构建包。> https://github.com/treasure-data/omnibus-td-agent