如果列出的次数超过X次，如何从文件中提取IP地址

Question

我想提取符合

1. 含foo
2. 在文件中列出超过5次

下面的日志示例

2020/12/07 03:25:16 [error] 31385#31385: *4283 limiting requests, excess: 100.110 by zone "foo", client: 1.1.1.1, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"
2020/12/07 03:25:16 [error] 31386#31386: *4107 limiting requests, excess: 100.962 by zone "bar", client: 1.1.1.2, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"
2020/12/07 03:25:16 [error] 31386#31386: *4107 limiting requests, excess: 100.962 by zone "bar", client: 1.1.1.2, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"
2020/12/07 03:25:16 [error] 31385#31385: *4164 limiting requests, excess: 100.102 by zone "foo", client: 1.1.1.1, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"
2020/12/07 03:25:16 [error] 31386#31386: *4107 limiting requests, excess: 100.962 by zone "bar", client: 1.1.1.2, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"
2020/12/07 03:25:16 [error] 31384#31384: *2404 limiting requests, excess: 100.080 by zone "foo", client: 1.1.1.1, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"
2020/12/07 03:25:16 [error] 31386#31386: *4107 limiting requests, excess: 100.962 by zone "bar", client: 1.1.1.2, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"
2020/12/07 03:25:16 [error] 31384#31384: *2321 limiting requests, excess: 100.062 by zone "foo", client: 1.1.1.1, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"
2020/12/07 03:25:16 [error] 31386#31386: *4107 limiting requests, excess: 100.962 by zone "bar", client: 1.1.1.2, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"
2020/12/07 03:25:16 [error] 31386#31386: *4220 limiting requests, excess: 100.020 by zone "foo", client: 1.1.1.1, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"
2020/12/07 03:25:16 [error] 31385#31385: *4406 limiting requests, excess: 100.002 by zone "foo", client: 1.1.1.1, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"
2020/12/07 03:25:16 [error] 31376#31376: *4172 limiting requests, excess: 100.996 by zone "foo", client: 1.1.1.1, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"
2020/12/07 03:25:16 [error] 31386#31386: *4190 limiting requests, excess: 100.988 by zone "foo", client: 1.1.1.1, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"
2020/12/07 03:25:16 [error] 31376#31376: *2549 limiting requests, excess: 100.984 by zone "foo", client: 1.1.1.1, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"
2020/12/07 03:25:16 [error] 31386#31386: *4189 limiting requests, excess: 100.972 by zone "foo", client: 1.1.1.1, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"
2020/12/07 03:25:16 [error] 31386#31386: *4107 limiting requests, excess: 100.962 by zone "bar", client: 1.1.1.2, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"
2020/12/07 03:25:16 [error] 31386#31386: *4107 limiting requests, excess: 100.962 by zone "bar", client: 1.1.1.2, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"
2020/12/07 03:25:16 [error] 31386#31386: *4107 limiting requests, excess: 100.962 by zone "foo", client: 1.1.1.1, server: example.com, request: "POST /some-link HTTP/1.1", host: "www.example.com", referrer: "https://www.example.com/some-link"

的结果应该是

1.1.1.1

1.1.1.2不应打印，因为它不是foo区域的一部分

我已经能够列出每个IP被列出的次数

grep -o "[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+" testfile | sort | uniq -c
     11 1.1.1.1
      7 1.1.1.2

但我不知道如何要求foo，然后将列出5次以上的I写入一个文件。

Answer 1

用GNU awk：

gawk '
  /zone "foo"/ && match($0, /client: ([0-9]+\.[0-9]+\.[0-9]+\.[0-9]+)/,m) {
    count[m[1]]++
  } 
  END {
    for (client in count) {if (count[client] > 5) print client}
  }
' testfile

或者使用米勒 (这更具体，因为它将条目作为分隔的key: value对对待，并分别将匹配限制为命名字段excess和client )

mlr --dkvp --fs ', ' --ps ': ' \
  filter '$excess =~ "zone \"foo\""' then \
  put -q '@count[$client] += 1; end {for (client in @count) {if (@count[client] > 5){print client}}}
' testfile

Answer 2

我更改了命令行，以获得每一行和单独的参数，在此之后，我只需检查第一个字段是否大于4

单行例

while read -r proc; do val=`echo "$proc" | cut -d' ' -f1 `; ip=`echo $proc | cut -d' ' -f2`; if [ $val -gt 4 ]; then echo $ip; fi ; done <<< `grep -o "[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+" ipList.txt | sort | uniq -c`

为了更好地理解语法，请多行：

while read -r proc
do
    val=`echo "$proc" | cut -d' ' -f1 `
    ip=`echo $proc | cut -d' ' -f2`
    if [ $val -gt 4 ]
    then
        echo $ip
    fi
done <<< `grep -o "[0-9]\+\.[0-9]\+\.[0-9]\+\.[0-9]\+" ipList.txt | sort | uniq -c`