Docker中的Logstash -将2个事件合并为1个事件

Question

我正在通过他们的官方映像在Docker中运行ElasticStack；但是，当我试图使用Logstash -聚合插件组合具有相同RequestID的事件时，我目前正在收到以下错误消息：

无法创建管道{：reason=>“找不到任何名为”聚合“的过滤器插件。您确定这是正确的吗?尝试加载聚合筛选插件会导致此错误:加载请求的名为类型筛选器聚合的插件时出现问题。错误: NameError NameError}

尽管如此，我也不能100%确定如何使用Logstash -聚合插件将以下示例事件组合成一个事件：

{
    "@t": "2017-10-16T20:21:35.0531946Z",
    "@m": "HTTP GET Request: \"https://myapi.com/?format=json&trackid=385728443\"",
    "@i": "29b30dc6",
    "Url": "https://myapi.com/?format=json&trackid=385728443",
    "SourceContext": "OpenAPIClient.Client",
    "ActionId": "fd683cc6-9e59-427f-a9f4-7855663f3568",
    "ActionName": "Web.Controllers.API.TrackController.TrackRadioLocationGetAsync (Web)",
    "RequestId": "0HL8KO13F8US6:0000000E",
    "RequestPath": "/api/track/radiourl/385728443"
}
{
    "@t": "2017-10-16T20:21:35.0882617Z",
    "@m": "HTTP GET Response: LocationAPIResponse { Location: \"http://sample.com/file/385728443/\", Error: null, Success: True }",
    "@i": "84f6b72b",
    "Response":
    {
        "Location": "http://sample.com/file/385728443/",
        "Error": null,
        "Success": true,
        "$type": "LocationAPIResponse"
    },
    "SourceContext": "OpenAPIClient.Client",
    "ActionId": "fd683cc6-9e59-427f-a9f4-7855663f3568",
    "ActionName": "Web.Controllers.API.TrackController.TrackRadioLocationGetAsync (Web)",
    "RequestId": "0HL8KO13F8US6:0000000E",
    "RequestPath": "/api/track/radiourl/385728443"
}

可以指导我如何正确地组合这些事件，如果聚合是正确的插件，为什么内置插件似乎不是Logstash映像的一部分？

docker-compose.yml内容：

 version: '3'
 services:
   elasticsearch:
     image: docker.elastic.co/elasticsearch/elasticsearch:5.6.3
     container_name: elasticsearch
     environment:
       - discovery.type=single-node
       - xpack.security.enabled=false
     ports:
       - 9200:9200
     restart: always
   logstash:
     image: docker.elastic.co/logstash/logstash:5.6.3
     container_name: logstash
     environment:
       - xpack.monitoring.elasticsearch.url=http://elasticsearch:9200
     depends_on:
       - elasticsearch
     ports:
       - 10000:10000
     restart: always
     volumes:
       - ./logstash/pipeline/:/usr/share/logstash/pipeline/
   kibana:
     image: docker.elastic.co/kibana/kibana:5.6.3
     container_name: kibana
     environment:
       - xpack.monitoring.elasticsearch.url=http://elasticsearch:9200
     depends_on:
       - elasticsearch
     ports:
       - 5601:5601
     restart: always

logstash/管道/empstore.conf目录：

 input {
     http {
         id => "empstore_http"
         port => 10000
         codec => "json"
     }
 }

 output {
     elasticsearch {
         hosts => [ "elasticsearch:9200" ]
         id => "empstore_elasticsearch"
         index => "empstore-openapi"
     }
 }

 filter {
     mutate {
         rename => { "RequestId" => "RequestID" }
     }

    aggregate {
         task_id => "%{RequestID}"
         code => ""
     }
 }

Answer 1

筛选器中的代码是必需的设置。

代码示例：

• Request_END： 代码=>“map‘’sql_event.get‘+= event.get(’D时长‘)”
• Request_START： 代码=>“map‘’sql_ code‘= 0”
• 请求： 代码=>“map‘’sql_event.get‘+= event.get(’D时长‘)”