分析nginx错误日志的grok模式的问题

Question

你好，我在日志文件中有下面一行

2018/05/11 23:08:28 [error] 53734#53734: *621532077 upstream prematurely closed connection while reading response header from upstream, client: 192.168.22.10, server: www.testserver.pt, request: "GET /methods/userinfo.ashx/getUserOpenBetsData? HTTP/2.0", upstream: "https://188.11.2.3:443/methods/userinfo.ashx/getUserOpenBetsData?", host: "www.testserver.pt", referrer: "https://www.testserver.pt/"

我正在尝试使用下面的grok来解析它

input {
    beats {
        port => "5044"
    }
}
 filter {
        grok{
        match => {"message" => '%{F_TIMESTAMP: timestamp} \[%{DATA:Message_type}\] %{DATA:EventId}\: \*%{NUMBER:Secondaryid} %{GREEDYDATA:Message}, client: %{IP:origin}, server: %{URIHOST:domain}, request: "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:httpversion}", upstream: %{QS:userRequest}, host: "%{URIHOST:host}", referrer: %{QS:referrer}'}
        }
        date{
        locale => "en"
        match => ["timestamp", "YYYY/MM/dd HH:mm:ss"]
        target => "@timestamp"
        }
}
output {
    elasticsearch {
        hosts => [ "localhost:9200" ]
        index => "logstash-%{+YYYY.MM.dd.HH}"
        user => "elastic"
        password => "changeme"

不是在耍把戏。

Answer 1

一个简单的Google搜索显示了它的NGINX日志，

您可以使用下面的grok模式，

(?<timestamp>%{YEAR}[./]%{MONTHNUM}[./]%{MONTHDAY} %{TIME}) \[%{LOGLEVEL:severity}\] %{POSINT:pid}#%{NUMBER:threadid}\: \*%{NUMBER:connectionid} %{GREEDYDATA:errormessage}, client: %{IP:client}, server: %{GREEDYDATA:server}, request: %{GREEDYDATA:request}

输出

{
  "timestamp": [
    [
      "2018/05/11 23:08:28"
    ]
  ],
  "YEAR": [
    [
      "2018"
    ]
  ],
  "MONTHNUM": [
    [
      "05"
    ]
  ],
  "MONTHDAY": [
    [
      "11"
    ]
  ],
  "TIME": [
    [
      "23:08:28"
    ]
  ],
  "HOUR": [
    [
      "23"
    ]
  ],
  "MINUTE": [
    [
      "08"
    ]
  ],
  "SECOND": [
    [
      "28"
    ]
  ],
  "severity": [
    [
      "error"
    ]
  ],
  "pid": [
    [
      "53734"
    ]
  ],
  "threadid": [
    [
      "53734"
    ]
  ],
  "BASE10NUM": [
    [
      "53734",
      "621532077"
    ]
  ],
  "connectionid": [
    [
      "621532077"
    ]
  ],
  "errormessage": [
    [
      "upstream prematurely closed connection while reading response header from upstream"
    ]
  ],
  "client": [
    [
      "192.168.22.10"
    ]
  ],
  "IPV6": [
    [
      null
    ]
  ],
  "IPV4": [
    [
      "192.168.22.10"
    ]
  ],
  "server": [
    [
      "www.testserver.pt"
    ]
  ],
  "request": [
    [
      ""GET /methods/userinfo.ashx/getUserOpenBetsData? HTTP/2.0", upstream: "https://188.11.2.3:443/methods/userinfo.ashx/getUserOpenBetsData?", host: "www.testserver.pt", referrer: "https://www.testserver.pt/""
    ]
  ]
}

你可以在这里测试一下。

还请查看github上的以下解析nginx错误日志的示例。

希望能帮上忙。