查找cc攻击IPs通过日志文件使用shell脚本

Question

我有这样的历史web日志文件：

157.15.14.19 - -  06 Sep 2016 09:13:10 +0300  "GET /index.php?id=1 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:13:11 +0300  "GET /index.php?id=2 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:13:12 +0300  "GET /index.php?id=3 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:14:13 +0300  "GET /index.php?id=4 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:14:14 +0300  "GET /index.php?id=5 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:15:15 +0300  "GET /index.php?id=6 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:15:16 +0300  "GET /index.php?id=7 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:15:17 +0300  "GET /index.php?id=8 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:16:10 +0300  "GET /index.php?id=9 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:16:10 +0300  "GET /index.php?id=10 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
8.8.8.8 - -  06 Sep 2016 09:17:10 +0300  "GET /index.php?id=11 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
9.9.9.9 - -  06 Sep 2016 09:17:10 +0300  "GET /index.php?id=12 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:18:10 +0300  "GET /index.php?id=13 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:19:10 +0300  "GET /index.php?id=14 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:19:10 +0300  "GET /index.php?id=15 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:20:10 +0300  "GET /index.php?id=15 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
123.123.123.123 - -  06 Sep 2016 09:21:10 +0300  "GET /index.php?id=15 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
157.15.14.19 - -  06 Sep 2016 09:22:10 +0300  "GET /index.php?id=15 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"

我只想通过昨天的web日志文件找到cc攻击的I。

在这个例子中，我签署了一个cc攻击：

每5分钟，相同的远程ip请求计数超过5，ip将受到cc攻击并打印出来。

日志文件是一整天的，只使用bash脚本，就像awk、cat、gawk、sed等等。

请给我一些建议，非常感谢。

更新：

我尝试使用测试脚本(每2分钟相同的请求count>5)

yy@yy:/tmp/tb$ cat 5.txt |awk '{print $7,$1}' |awk -F: '{print $1*60+int($2/2),$0}' |sort |uniq -c -f2 |awk '{if($1>5){print $0}}'
     10 546 09:13:10 157.15.14.19

但是，代码是如此糟糕，它将是优化的。

Answer 1

awk -v Interval=5 -v Trig=5 -F '[[:blank:]]*|:' '
        {
        # using format log
        #  157.15.14.19 - -  06 Sep 2016 09:13:10 +0300  "GET /index.php?id=1 HTTP/1.1" 200 16977 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)"
        # $1            2 3  4  5   6    7  8  9  10      11 ...

        ThisTime = $7 * 60 + $8
        #if new cycle (so this line is not in the cycle)
        if ( ThisTime > ( LastTic + Interval ) ) {
          # check and print last cycle hit
          for( IP in IPCounts) if ( IPCounts[ IP] > Trig) print LastTime " " IP " : " IPCounts[ IP]

          # reset reference
          split( "", IPCounts)
          LastTime = $4 " " $5 " " $6 " " $7 ":" sprintf( "%2d", ( $8 - ( $8 % Interval) )) ":00"
          LastTic = $7 * 60 + ( $8 - ( $8 % Interval) )
          }
        # add this line to new cycle
        IPCounts[ $1]++
        }

        END {
          # print last cycle
          for( IP in IPCounts) if ( IPCounts[ IP] > Trig) print LastTime " " IP " : " IPCounts[ IP]
          }
      ' YourFile


# for format of log
#  op.g.cc 124.145.36.121 - - [21/Nov/2016:03:38:02 +0800] ==> 172.11.0.238:80 "POST ...
# $1       2              3 4 5            6  7  8  9      10   11 ...  

# change:
#  $7 by $6, $8 by $7
#  LastTime = $5 ":" $6 ":" sprintf( "%2d", ( $7 - ( $7 % Interval) )) ":00 +800]"
#  IPCounts[ $2]++

注意：

• 快速和肮脏的时间选择(你提到每天一条日志)。如果需要更高的精度，请使用mkftime来使用实际的时代时间基准。
• Trig是计数触发级别(5次)，间隔是周期的时间(5分钟)。