gunicorn访问日志格式

Question

我打算在库伯内斯通过黑角兽查一查。为了获得正确的日志记录，我希望以json格式输出所有日志。

目前，我正在使用minikube和https://github.com/inovex/kubernetes-logging进行测试，以便让fluentd收集日志。

我设法使错误日志(回溯)格式正确，这要归功于：JSON formatted logging with Flask and gunicorn

我还在为访问日志格式而苦苦挣扎。我指定了以下gunicorn访问日志格式：

access_log_format = '{"remote_ip":"%(h)s","request_id":"%({X-Request-Id}i)s","response_code":"%(s)s","request_method":"%(m)s","request_path":"%(U)s","request_querystring":"%(q)s","request_timetaken":"%(D)s","response_length":"%(B)s"}'

并且结果日志是json格式的。但是消息部分(基于access_log_format的格式)现在包含转义的双引号，并且不会被fluentd / ELK解析到它的单独字段中

{"tags": [], "timestamp": "2017-12-07T11:50:20.362559Z", "level": "INFO", "host": "ubuntu", "path": "/usr/local/lib/python2.7/dist-packages/gunicorn/glogging.py", "message": "{\"remote_ip\":\"127.0.0.1\",\"request_id\":\"-\",\"response_code\":\"200\",\"request_method\":\"GET\",\"request_path\":\"/v1/records\",\"request_querystring\":\"\",\"request_timetaken\":\"19040\",\"response_length\":\"20\"}", "logger": "gunicorn.access"}

感谢Jpw

Answer 1

最简单的解决方案是将外部单引号更改为双引号，将内部双引号更改为单引号，如下所述。

--access-logformat  "{'remote_ip':'%(h)s','request_id':'%({X-Request-Id}i)s','response_code':'%(s)s','request_method':'%(m)s','request_path':'%(U)s','request_querystring':'%(q)s','request_timetaken':'%(D)s','response_length':'%(B)s'}"

以下是示例日志

{'remote_ip':'127.0.0.1','request_id':'-','response_code':'404','request_method':'GET','request_path':'/test','request_querystring':'','request_timetaken':'6642','response_length':'233'}
{'remote_ip':'127.0.0.1','request_id':'-','response_code':'200','request_method':'GET','request_path':'/','request_querystring':'','request_timetaken':'881','response_length':'20'}

Answer 2

您可以在JSON的值中直接转义双引号(\")，以使您的日志保持有效的--access-logformat。

因此，如果您在Docker容器中运行Gunicorn，您的Dockerfile可能会以如下形式结束：

CMD ["gunicorn",            \
    "-b", "0.0.0.0:5000",   \
    "--access-logfile", "-",\
    "--access-logformat", "{\"remote_ip\":\"%(h)s\",\"request_id\":\"%({X-Request-Id}i)s\",\"response_code\":\"%(s)s\",\"request_method\":\"%(m)s\",\"request_path\":\"%(U)s\",\"request_querystring\":\"%(q)s\",\"request_timetaken\":\"%(D)s\",\"response_length\":\"%(B)s\"}", \
    "app:create_app()"]

找到Gunicorn日志记录选项here的其余部分。

Answer 3

我在一个日志配置文件中寻找一些有用的东西。此外，我也不喜欢手动构建json格式。

因此，让我们从那里获取字段，并让pythonjsonlogger为我们完成其余的工作。

格式化程序类

from pythonjsonlogger.jsonlogger import JsonFormatter, merge_record_extra


class GunicornLogFormatter(JsonFormatter): 
    def add_fields(self, log_record, record, message_dict):
        """
        This method allows us to inject gunicorn's args as fields for the formatter
        """
        super(GunicornLogFormatter, self).add_fields(log_record, record, message_dict)
        for field in self._required_fields:
            if field in self.rename_fields:
                log_record[self.rename_fields[field]] = record.args.get(field)
            else:
                log_record[field] = record.args.get(field)

日志记录配置文件示例

version: 1
disable_existing_loggers: False
formatters:
  gunicorn_json:
    (): GunicornLogFormatter
    format: '%(h)s %(r)s %({x-request-id}i)s'
    datefmt: '%Y-%m-%dT%H:%M:%S%z'
    rename_fields:
      levelname: level
      '{x-request-id}i': request_id
      r: request

handlers:
  json-gunicorn-console:
    class: logging.StreamHandler
    level: INFO
    formatter: gunicorn_json
    stream: ext://sys.stdout

loggers:
  gunicorn.access:
    level: INFO
    handlers:
      - json-gunicorn-console

示例日志

{"h": "127.0.0.1", "request": "GET /login?next=/ HTTP/1.1", "request_id": null}
{"h": "127.0.0.1", "request": "GET /static/css/style.css HTTP/1.1", "request_id": null}
{"h": "127.0.0.1", "request": "GET /some/random/path HTTP/1.1", "request_id": null}
{"h": "127.0.0.1", "request": "GET /some/random/path HTTP/1.1", "request_id": "123123123123123123"}