Filebeat decode_json_fields没有解析数组

Question

我们使用filbeat处理器decode_ Json -field来处理Json中的日志消息。我们遇到的问题是，我们的一些日志是多层的，有相当多的数组和一些嵌套对象。我们尝试在process_array标志设置为true的情况下使用decode_json_fields，但是Filebeat仍然将'[‘后面的所有内容都放在一个字段中。

这是我们在Kibana的显示屏上看到的：

@timestamp                   Oct 28, 2019 @ 12:22:06.610
t _id                        pKEaEm4B7zyLz8s9M8Xe
t _index                     filebeat-7.3.2-2019.10.28-000001
# _score                     -
t _type                      _doc
t agent.ephemeral_id         7c3cd7b7-2f76-424e-a417-5aa82f119bed
t agent.hostname             ******
t agent.id                   571154fa-e864-49b1-a224-9d405befeddf
t agent.type                 filebeat
t agent.version              7.3.2
? circuitPath                { "policy": "Health Check LB", "execTime": 0, "filters": [ { "class": "com.vordel.circuit.attribute.CompareAttributeFilter", "status": "Pass", "filterTime": 1557733771853, "execTime": 0, "espk": "PrimaryStore-43595d15-05f6-4135-aa9a-e8b9b2a35bda:-439438454261778670", "name": "Compare Attribute", "type": "CompareAttributeFilter" }, { "execTime": 0, "espk": "PrimaryStore-43595d15-05f6-4135-aa9a-e8b9b2a35bda:-6704867506249825459", "name": "Set Message - OK", "type": "ChangeMessageFilter", "class": "com.vordel.circuit.conversion.ChangeMessageFilter", "status": "Pass", "filterTime": 1557733771853 }, { "execTime": 0, "espk": "PrimaryStore-43595d15-05f6-4135-aa9a-e8b9b2a35bda:-5308572925601299001", "name": "Reflect - OK", "type": "ReflectFilter", "class": "com.vordel.circuit.net.ReflectFilter", "status": "Pass", "filterTime": 1557733771853 } ] }
? correlationId              *******************
t ecs.version                1.0.1
t host.name                  *****
t input.type                 log
t log.file.path              *****
# log.offset                 747,788
? processInfo.domainId       *******************
? processInfo.groupId        group-2
? processInfo.groupName      ******
? processInfo.hostname       f3slsea310
? processInfo.serviceId      instance-6
? processInfo.serviceName    ******
? processInfo.version        7.6.2 SP1
suricata.eve.timestamp       Oct 28, 2019 @ 12:22:06.610
? timestamp                  1557733771854

正如您所看到的，当Filebeat到达嵌套数组"circuitPath“时，它会解析单个字段中的所有内容，直到数组关闭。

以下是我们遇到问题的一个日志的示例：

{"timestamp":1557733646862,"correlationId":"***************","processInfo":{"hostname":"f3slsea310","domainId":"*******************","groupId":"group-2","groupName":"*****","serviceId":"instance-6","serviceName":"*******","version":"7.6.2 SP1"},"circuitPath":[ { "policy": "Health Check LB", "execTime": 0, "filters": [  { "espk": "PrimaryStore-43595d15-05f6-4135-aa9a-e8b9b2a35bda:-439438454261778670", "name": "Compare Attribute", "type": "CompareAttributeFilter", "class": "com.vordel.circuit.attribute.CompareAttributeFilter", "status": "Pass", "filterTime": 1557733646861, "execTime": 0 } , { "espk": "PrimaryStore-43595d15-05f6-4135-aa9a-e8b9b2a35bda:-6704867506249825459", "name": "Set Message - OK", "type": "ChangeMessageFilter", "class": "com.vordel.circuit.conversion.ChangeMessageFilter", "status": "Pass", "filterTime": 1557733646861, "execTime": 0 } , { "espk": "PrimaryStore-43595d15-05f6-4135-aa9a-e8b9b2a35bda:-5308572925601299001", "name": "Reflect - OK", "type": "ReflectFilter", "class": "com.vordel.circuit.net.ReflectFilter", "status": "Pass", "filterTime": 1557733646861, "execTime": 0 }  ] } ]}

Filebeat.yml

processors:
   - decode_json_fields:
       fields: [message]
       process_array: true
       max_depth: 11
       overwrite_keys: true

Answer 1

您必须指定“​​”指令，以便filebeat知道删除哪个字段中的值，或将其保留为空- "“，以便所有这些值目标都转到根目录

https://www.elastic.co/guide/en/beats/filebeat/current/decode-json-fields.html