弹性搜索中基于字段数据的索引

Question

我正在使用logstash (版本1.5.3)导入一个syslog到elasticsearch(版本1.7.1)，使用以下配置。

input{
  file {
    path => "somepath\*.log"
  }
}
filter{
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:message_hostname} %{DATA:message_program}(?:\[%{POSINT:message_pid}\])?: %{GREEDYDATA:user_message}" }
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
}
output {
  elasticsearch{
    cluster => somecluster 
    host => localhost 
    index => "logindex-%{+YYYY-MM-dd}"
  }
}

我的索引是根据当前日期和时间(即logindex-2015-08-07 )创建的。

我希望使用上述格式( syslog_timestamp )创建索引，而不是根据当前日期创建索引。

因此，如果日志具有时间戳2015-01-01，则应该将我的索引创建为logindex-2015-01-01而不是logindex-2015-08-07。

编辑

使用的日志输入：

Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message

日志调试输出

←[36mfilter received {:event=>{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]:(root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT"}, :level=>:debug, :file=>"(eval)", :line=>"69", :method=>"filter_func"}
 ←[0m
 ←[36mRunning grok filter {:event=>#<LogStash::Event:0x166d6250 @metadata_accessors=#<LogStash::Util::Accessors:0x14fe0ed7 @store={}, @lut={}>, @cancelled=false, @data={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT"}, @metadata={}, @accessors=#<LogStash::Util::Accessors:0x6ee8ffaf @store={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT"}, @lut={"host"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root)
CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT"}, "host"]}>>, :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-1.0.0/lib/logstash/filters/grok.rb", :line=>"283", :method=>"filter"}
←[0m
←[36mRegexp match object {:names=>["SYSLOGTIMESTAMP:message_timestamp", "SYSLOGHOST:message_hostname", "DATA:message_program", "POSINT:message_pid", "GREEDYDATA:user_message"], :captures=>["Jul 27 07:49:01", "Server1", "CRON", "21009", "(root) CMD LTest Message\r"], :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.2/lib/grok-pure.rb", :line=>"179", :method=>"match_and_capture"}
←[0m
←[36mfilters/LogStash::Filters::Grok: adding value to field {:field=>"received_at", :value=>["%{@timestamp}"], :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.3-java/lib/logstash/util/decorators.rb", :line=>"28", :method=>"add_fields"}

←[0m
←[36mEvent now:  {:event=>#<LogStash::Event:0x166d6250 @metadata_accessors=#<LogStash::Util::Accessors:0x14fe0ed7 @store={}, @lut={}>, @cancelled=false, @data={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, @metadata={}, @accessors=#<LogStash::Util::Accessors:0x6ee8ffaf @store={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, @lut={"host"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "host"], "message"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009","user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message"], "message_timestamp"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_timestamp"], "message_hostname"=>[{"message"=>"Jul27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_hostname"], "message_program"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTestMessage\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_program"], "message_pid"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_pid"], "user_message"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "user_message"], "@timestamp"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "@timestamp"], "received_at"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01","message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "received_at"]}>>, :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-1.0.0/lib/logstash/filters/grok.rb", :line=>"303", :method=>"filter"}

←[0m
←[36mDate filter: received event {:type=>nil, :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/logstash-filter
-date-1.0.0/lib/logstash/filters/date.rb", :line=>"206", :method=>"filter"}
←[0m←[36mDate filter looking for field {:type=>nil, :field=>"syslog_timestamp", :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/
jruby/1.9/gems/logstash-filter-date-1.0.0/lib/logstash/filters/date.rb", :line=>
"209", :method=>"filter"}
←[0m
←[36moutput received {:event=>{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]:(root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, :level=>:debug, :file=>"(eval)", :line=>"76", :method=>"output_func"}
←[0m
←[36mFlushing output {:outgoing_count=>1, :time_since_last_flush=>22.048, :outgoing_events=>{nil=>[["index", {:_id=>nil, :_index=>"%index-2015-08-14", :_type=>"logs", :_routing=>nil},
#<LogStash::Event:0x166d6250 
@metadata_accessors=#<LogStash::Util::Accessors:0x14fe0ed7 @store={"retry_count"=>0}, @lut={}>, @cancelled=false, @data={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, @metadata={"retry_count"=>0}, 
/*
* 提示：该行代码过长，系统自动注释不进行高亮。一键复制会移除系统注释 
* @accessors=#<LogStash::Util::Accessors:0x6ee8ffaf @store={"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, @lut={"host"=>[{"message"=>sage\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, @lut={"host"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r","received_at"=>"2015-08-14T07:34:53.215Z"}, "host"], "message"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message"], "message_timestamp"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_timestamp"], "message_hostname"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_hostname"], "message_program"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_program"], "message_pid"=>[{"message"=>"Jul 27 07:49:01 Server1CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "message_pid"], "user_message"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "user_message"], "@timestamp"=>[{"message"=>"Jul 2707:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "@timestamp"], "received_at"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "received_at"], "type"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r","received_at"=>"2015-08-14T07:34:53.215Z"}, "type"], "syslog_timestamp"=>[{"message"=>"Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r", "@version"=>"1", "@timestamp"=>"2015-08-14T07:34:53.215Z", "host"=>"HOST-LT", "message_timestamp"=>"Jul 27 07:49:01", "message_hostname"=>"Server1", "message_program"=>"CRON", "message_pid"=>"21009", "user_message"=>"(root) CMD LTest Message\r", "received_at"=>"2015-08-14T07:34:53.215Z"}, "syslog_timestamp"]}>>]]}, :batch_timeout=>1, :force=>nil, :final=>nil, :level=>:debug, :file=>"/Folder/logstash-1.5.3/logstash-1.5.3/vendor/bundle/jruby/1.9/gems/stud-0.0.20/lib/stud/buffer.rb", :line=>"207", :method=>"buffer_flush"}
*/

←[0m{              "message" => "Jul 27 07:49:01 Server1 CRON[21009]: (root) CMD LTest Message\r",
             "@version" => "1",
           "@timestamp" => "2015-08-14T07:34:53.215Z",
                 "host" => "HOST-LT",
    "message_timestamp" => "Jul 27 07:49:01",
     "message_hostname" => "Server1",
      "message_program" => "CRON",
          "message_pid" => "21009",
         "user_message" => "(root) CMD LTest Message\r",
          "received_at" => "2015-08-14T07:34:53.215Z"
}

Answer 1

问题可能是计算机上的区域设置与用于生成日志的区域设置不同。因此，您应该在locale过滤器中指定date，如下所示：

...
filter{
    grok {
        ...
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
      locale => "en"
    }
}
...

更新

基于上面的日志输出(非常有用！！)，date过滤器应该在message_timestamp字段上工作，而不是syslog_timestamp (不存在)。