无法使用syslog-ng OSE 3.33从RFC6587解析PulseSecure框架内的syslog

Question

PulseSecure设备正在发送符合RFC5424的syslog。当使用TCP作为传输时，RFC6587帧将预先添加到syslog消息(MSG_LEN SP SYSLOG_MSG)中。我还需要在相同的TCP端口上接收syslog，而不需要RFC6587框架，因此syslog源代码不是一个可以使用的选项，因为它期望框架在任何时候都存在。我尝试的是使用网络驱动程序和禁用使用标志的解析(不解析)，并有一个重写规则来删除RFC6587帧。重写规则确实有效--框架确实被删除了。但是，当我尝试用syslog解析器解析消息时，它似乎不起作用；例如，没有提取主机名(使用IP地址)。我尝试使用正则表达式匹配来识别RFC5424或RFC3164 syslog，这也不起作用；它总是以if结构的“否则”部分结束。即使删除if结构也没有帮助(强制使用rfc5424解析器)。

测试配置如下(我使用TCP/602进行测试)：

source s_net_tcp602 {
  network(
    port(602)
    transport("tcp")
    flags(no-parse)           # Store entire message into MESSAGE/MSG macro
    flags(store-raw-message)  # Store original message in RAWMSG macro
  );
};

# RFC6587 framing: MSG_LEN SP SYSLOG-MSG
# This rewrite rule removes the MSG_LEN SP completely from MESSAGE macro if present
rewrite rw_remove_rfc6587_framing {
  subst(
    "^\d+\s+<",      # Matches digits followed by one or more spaces and '<' bracket  at start of MESSAGE
    "<",             # Replace with only '<' bracket so effectively RFC6587 framing removed
    value("MESSAGE")
  );
};

parser p_syslog_rfc3164 {
  syslog-parser(
    drop-invalid(no)
    flags(validate-utf8)
    template("${MESSAGE}")
  );
};

parser p_syslog_rfc5424 {
  syslog-parser(
    flags(syslog-protocol)
    drop-invalid(no)
    flags(validate-utf8)
    template("${MESSAGE}")
  );
};

destination d_test_5424 { file("/var/log/test5424/${S_YEAR}_${S_MONTH}_${S_DAY}_${HOST}.log" template("${MESSAGE}\n") perm(0644) create_dirs(yes)); };
destination d_test_3164 { file("/var/log/test3164/${S_YEAR}_${S_MONTH}_${S_DAY}_${HOST}.log" template("${MESSAGE}\n") perm(0644) create_dirs(yes)); };

log {

  # Complete message should be stored in MESSAGE and RAWMSG macro
  source(s_net_tcp602);

  # Strip MSG_LEN part from MESSAGE if it matches RFC6587 framing and continue
  rewrite(rw_remove_rfc6587_framing);

  # MESSAGE macro now contains either RFC3164 or RFC5424 formatted message

  # Use If mechanism to determine if MESSAGE conforms to RFC5424 or RFC3164
  # Using two different destinations to allow determining result of regular expression matching
  if (message('^<\d+>\d+\s')) {
    parser(p_syslog_rfc5424);
    destination(d_test_5424);
  } else {
    parser(p_syslog_rfc3164);
    destination(d_test_3164);
  };

};

来自PulseSecure的syslog消息示例：

196 <134>1 2022-03-29T10:38:05+02:00 hostname PulseSecure: - - - 2022-03-29 10:38:05 - ive - <REMOVED MSG>

这可能是我犯的一个明显的错误，希望有人能指出正确的方向.

提前谢谢你，马可

我尝试使用重写规则删除RFC6587框架，但之后syslog解析器没有成功地解析消息。我还试图将syslog从另一个设备( CitrixADC )发送到同一个目的地，并且似乎尝试了解析，但失败了，因为CitrixADC syslog可能并不完全符合RFC3164。试图完全删除if-else结构，并强迫RFC5424解析没有成功。

Answer 1

我似乎能够使用带有专用过滤器的连接/通道来识别RFC3164或RFC5424格式的syslog来解决这个问题。仍然在进行测试，但是主机名现在已从RFC5424格式化的syslog中正确解析，而RFC6587框架则来自PulseSecure设备。

当前的示例配置似乎有效：

source s_net_tcp602 {
  network(
    port(602)
    transport("tcp")
    flags(no-parse)           # Store entire message into MESSAGE/MSG macro
    flags(store-raw-message)  # Store original message in RAWMSG macro
  );
};

# RFC6587 framing: MSG_LEN SP SYSLOG-MSG
# This rewrite rule removes the MSG_LEN SP completely from MESSAGE macro if present
rewrite rw_remove_rfc6587_framing {
  subst(
    '^\d+\s+<',      # Matches digits followed by one or more spaces and '<' bracket  at start of MESSAGE
    '<',             # Replace with only '<' bracket so effectively RFC6587 framing removed
    value("MESSAGE"),
    flags("utf8")
  );
};

# Matches messages formatted according to RFC5424:
# PRI VERSION SP TIMESTAMP HOSTNAME SP APP-NAME SP PROCID SP MSGID SP STRUCTURED-DATA [SP MSG]
#   PRI is 1-3 digits between '<' and '>' brackets
#   VERSION is 0-2 digits so seems not to be mandatory? ==> We assume 1 or 2 digits
#   SP is a single whitespace
#
# Example start of message:
# <134>1 2022-03-29T14:33:18+02:00 hostname ...
filter f_rfc5424 {
  # PCRE-style regular expression
  # Using single quotes to prevent interpretation
  match('^<\d{1,3}>\d{1,2}\s' value("MESSAGE"));
};

# Matches messages formatted according to RFC3164:
# PRI TIMESTAMP SP HOSTNAME SP MSG
#   PRI       1-3 digits between '<' and '>' brackets
#   TIMESTAMP Local time in format "Mmm dd hh:mm:ss" (without the quotes)
#
# Example start of message:
# <134>Mar 29 14:33:18 hostname ...
filter f_rfc3164 {
  # PCRE-style regular expression
  # Using single quotes to prevent interpretation
  match('^<\d{1,3}>\D{3}\s' value("MESSAGE"));
};

parser p_syslog_rfc3164 {
  syslog-parser(
    drop-invalid(no)
    flags(validate-utf8)
    flags(sanitize-utf8)
    template("${MESSAGE}")
  );
};

parser p_syslog_rfc5424 {
  syslog-parser(
    flags(syslog-protocol)
    drop-invalid(no)
    flags(validate-utf8)
    flags(sanitize-utf8)
    template("${MESSAGE}")
  );
};

destination d_test_pre-parse { file("/var/log/test/preparse_${S_YEAR}_${S_MONTH}_${S_DAY}_${HOST}.log" template("${MESSAGE}\n") perm(0644) create_dirs(yes)); };
destination d_test_rfc5424 { file("/var/log/test/rfc5424_${S_YEAR}_${S_MONTH}_${S_DAY}_${HOST}.log" template("${RAWMSG}\n") perm(0644) create_dirs(yes)); };
destination d_test_rfc3164 { file("/var/log/test/rfc3164_${S_YEAR}_${S_MONTH}_${S_DAY}_${HOST}.log" template("${RAWMSG}\n") perm(0644) create_dirs(yes)); };
destination d_test_nonrfc { file("/var/log/test/nonrfc_${S_YEAR}_${S_MONTH}_${S_DAY}_${HOST}.log" template("${RAWMSG}\n") perm(0644) create_dirs(yes)); };

log {

  # Complete message should be stored in MESSAGE and RAWMSG macro as result of this source
  source(s_net_tcp602);

  # Strip MSG_LEN part from MESSAGE if it matches RFC6587 framing and continue
  rewrite(rw_remove_rfc6587_framing);

  # MESSAGE macro now contains either RFC3164 or RFC5424 formatted message
  destination(d_test_pre-parse);    # Just for testing, to be removed when done

  junction {
    # RFC5424 syslog
    channel {
      filter(f_rfc5424);
      parser(p_syslog_rfc5424);
      destination(d_test_rfc5424);
      flags(final);
    };
    # RFC3164 syslog
    channel {
      filter(f_rfc3164);
      parser(p_syslog_rfc3164);
      destination(d_test_rfc3164);
      flags(final);
    };
    # Non-RFC syslog
    channel {
      destination(d_test_nonrfc);
    };
  };
};