如何在Fluentd配置中定义散列(JSON或object)？

Question

我有一个问题，格式标签的谷歌云插件的Fluentd。我想用我自己的标签设置logging.googleapis.com/labels字段。这就是Google插件文档对这个领域的看法：

该字段的值应该是结构化记录。

Fluentd文件提到有一个hash数据类型：

hash：该字段被解析为JSON对象。它还支持速记语法。这些都是相同的价值：

• 正常：{"key1": "value1", "key2": "value2"}
• 速记：key1:value1,key2:value2

因此，我尝试了这个(引号+大括号)：

<filter app.**>
  @type record_transformer

  renew_record true

  <record>
    severity ${record["severity"]}
    message ${record["message"]}
    logging.googleapis.com/trace ${record["trace"]}
    logging.googleapis.com/spanId ${record["spanId"]}
    logging.googleapis.com/labels {
      "kubernetes-host": "${record.dig("kubernetes", "host")}",
      "kubernetes-pod-name": "${record.dig("kubernetes", "pod_name")}",
      "kubernetes-pod-id": "${record.dig("kubernetes", "pod_id")}",
      "kubernetes-pod-ip": "${record.dig("kubernetes", "pod_ip")}",
      "kubernetes-container-name": "${record.dig("kubernetes", "container_name")}",
      "kubernetes-namespace-name": "${record.dig("kubernetes", "namespace_name")}",
      "kubernetes-namespace-id": "${record.dig("kubernetes", "namespace_id")}"
    }
  </record>
</filter>

我也尝试过这样做(没有引号，但使用了大括号)：

<filter app.**>
  @type record_transformer

  renew_record true

  <record>
    severity ${record["severity"]}
    message ${record["message"]}
    logging.googleapis.com/trace ${record["trace"]}
    logging.googleapis.com/spanId ${record["spanId"]}
    logging.googleapis.com/labels {
      "kubernetes-host": ${record.dig("kubernetes", "host")},
      "kubernetes-pod-name": ${record.dig("kubernetes", "pod_name")},
      "kubernetes-pod-id": ${record.dig("kubernetes", "pod_id")},
      "kubernetes-pod-ip": ${record.dig("kubernetes", "pod_ip")},
      "kubernetes-container-name": ${record.dig("kubernetes", "container_name")},
      "kubernetes-namespace-name": ${record.dig("kubernetes", "namespace_name")},
      "kubernetes-namespace-id": ${record.dig("kubernetes", "namespace_id")}
    }
  </record>
</filter>

我也尝试过这样做(没有引号，没有大括号)：

<filter app.**>
  @type record_transformer

  renew_record true

  <record>
    severity ${record["severity"]}
    message ${record["message"]}
    logging.googleapis.com/trace ${record["trace"]}
    logging.googleapis.com/spanId ${record["spanId"]}
    logging.googleapis.com/labels {
      "kubernetes-host": record.dig("kubernetes", "host"),
      "kubernetes-pod-name": record.dig("kubernetes", "pod_name"),
      "kubernetes-pod-id": record.dig("kubernetes", "pod_id"),
      "kubernetes-pod-ip": record.dig("kubernetes", "pod_ip"),
      "kubernetes-container-name": record.dig("kubernetes", "container_name"),
      "kubernetes-namespace-name": record.dig("kubernetes", "namespace_name"),
      "kubernetes-namespace-id": record.dig("kubernetes", "namespace_id")
    }
  </record>
</filter>

但是每次Fluentd失败时都会出现以下错误：

/opt/bitnami/fluentd/gems/fluentd-1.12.0/lib/fluent/config/basic_parser.rb:92:in `parse_error！：在fluentd.conf第92,9行获得不完整的JSON散列配置(fluentd.conf：：ConfigParseError)

我的整个配置(错误消息中的第92行是配置中使用</match>的最后一行)：

# Ignore fluentd own events
<match fluent.**>
  @type null
</match>

# TCP input to receive logs from
<source>
  @type forward

  bind 0.0.0.0
  port 24224
</source>

# HTTP input for the liveness and readiness probes
<source>
  @type http

  bind 0.0.0.0
  port 9880
</source>

# Throw the healthcheck to the standard output instead of forwarding it
<match fluentd.healthcheck>
  @type stdout
</match>

<filter kubernetes.var.log.containers.**.log>
  @type grep

  <regexp>
    key log
    pattern /^\[(?<logtime>[0-9T:.+-]+)\] \[(?<trace>.*?)\/(?<spanId>.*?)\] \[(?<channel>.+?)\] (?<severity>[A-Z]+): (?<message>.+)$/
  </regexp>
</filter>

<filter kubernetes.var.log.containers.**.log>
  @type parser

  key_name log

  <parse>
    @type regexp

    expression /^\[(?<logtime>[0-9T:.+-]+)\] \[(?<trace>.*?)\/(?<spanId>.*?)\] \[(?<channel>.+?)\] (?<severity>[A-Z]+): (?<message>.+)$/
    time_key logtime
    time_format %Y-%m-%dT%H:%M:%S.%L%z
  </parse>
</filter>

<match kubernetes.var.log.containers.**.log>
  @type rewrite_tag_filter

  <rule>
    key channel
    pattern /^(\w+)$/
    tag app.$1
  </rule>
</match>

<filter app.**>
  @type record_transformer

  renew_record true

  <record>
    severity ${record["severity"]}
    message ${record["message"]}
    logging.googleapis.com/trace ${record["trace"]}
    logging.googleapis.com/spanId ${record["spanId"]}
    logging.googleapis.com/labels {
      "kubernetes-host": record.dig("kubernetes", "host"),
      "kubernetes-pod-name": record.dig("kubernetes", "pod_name"),
      "kubernetes-pod-id": record.dig("kubernetes", "pod_id"),
      "kubernetes-pod-ip": record.dig("kubernetes", "pod_ip"),
      "kubernetes-container-name": record.dig("kubernetes", "container_name"),
      "kubernetes-namespace-name": record.dig("kubernetes", "namespace_name"),
      "kubernetes-namespace-id": record.dig("kubernetes", "namespace_id")
    }
  </record>
</filter>

<match app.**>
  @type google_cloud

  autoformat_stackdriver_trace true

  <inject>
    time_key time
    time_type string
    time_format %Y-%m-%dT%H:%M:%S.%NZ
  </inject>
</match>

如何正确定义此值？我没有找到任何使用hash数据类型(使用Ruby作为值)和在Google插件中使用logging.googleapis.com/labels属性的例子。

Answer 1

刚刚用这个测试过(双引号中的单引号)：

<filter app.**>
  @type record_transformer

  renew_record true

  <record>
    severity ${record["severity"]}
    message ${record["message"]}
    logging.googleapis.com/trace ${record["trace"]}
    logging.googleapis.com/spanId ${record["spanId"]}

    logging.googleapis.com/labels {
      "kubernetes-host": "${record.dig('kubernetes', 'host')}",
      "kubernetes-pod-name": "${record.dig('kubernetes', 'pod_name')}",
      "kubernetes-pod-id": "${record.dig('kubernetes', 'pod_id')}",
      "kubernetes-pod-ip": "${record.dig('kubernetes', 'pod_ip')}",
      "kubernetes-container-name": "${record.dig('kubernetes', 'container_name')}",
      "kubernetes-namespace-name": "${record.dig('kubernetes', 'namespace_name')}",
      "kubernetes-namespace-id": "${record.dig('kubernetes', 'namespace_id')}"
    }
  </record>
</filter>

对于--dry-run (对于不可用插件有一些注释配置)：

$ fluentd -c ./fluent-hash-test.conf --dry-run
2021-01-14 17:57:13 +0500 [info]: parsing config file is succeeded path="./fluent-hash-test.conf"
...
2021-01-14 17:57:13 +0500 [info]: using configuration file: <ROOT>
  <source>
    @type http
    bind "0.0.0.0"
    port 9880
  </source>
  <filter app.**>
    @type record_transformer
    renew_record true
    <record>
      severity ${record["severity"]}
      message ${record["message"]}
      logging.googleapis.com/trace ${record["trace"]}
      logging.googleapis.com/spanId ${record["spanId"]}
      logging.googleapis.com/labels {"kubernetes-host":"${record.dig('kubernetes', 'host')}","kubernetes-pod-name":"${record.dig('kubernetes', 'pod_name')}","kubernetes-pod-id":"${record.dig('kubernetes', 'pod_id')}","kubernetes-pod-ip":"${record.dig('kubernetes', 'pod_ip')}","kubernetes-container-name":"${record.dig('kubernetes', 'container_name')}","kubernetes-namespace-name":"${record.dig('kubernetes', 'namespace_name')}","kubernetes-namespace-id":"${record.dig('kubernetes', 'namespace_id')}"}
    </record>
  </filter>
</ROOT>
2021-01-14 17:57:13 +0500 [info]: finished dry run mode