计算Elasticsearch中滑动时间窗口的更改百分比

Question

Elasticsearch新手在这里。我有一系列这样的日志消息

{
  "@timestamp": "whatever",
  "type": "toBeMonitored",
  "success": true
}

与昨天相同的时间间隔相比，我的任务是对成功消息总数的-30%的变化做出反应。因此，如果我在今天上午8点进行检查，我应该将今天午夜到上午8点的总数与昨天的相同时间间隔进行比较。

我尝试创建日期直方图聚合，但我希望将diff百分比作为查询结果，而不是在开发端进行计算。

{
  "size": 0, 
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "type": "toBeMonitored"
          }
        },
        {
          "term": {
            "status": true
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "now-1d/d",
              "lte": "now/h"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "histo": {
       "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "1h"
      }
    }
  }
}

你知道如何实现这一点吗？

Answer 1

最终放弃了date_histogram聚合，而使用了date_range聚合。使用它要容易得多，即使它不会返回与昨天相同时间段的差值。我是用代码实现的。

{
  "size": 0,
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "type": "toBeMonitored"
          }
        },
        {
          "term": {
            "status": true
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "now-1d/d",
              "lte": "now/h"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "ranged_documents": {
      "date_range": {
        "field": "@timestamp",
        "ranges": [
          {
            "key": "yesterday",
            "from": "now-1d/d",
            "to": "now-24h/h"
          },
          {
            "key": "today",
            "from": "now/d",
            "to": "now/h"
          }
        ],
        "keyed": true
      }
    }
  }
}

此查询将产生类似于以下内容的结果

{
    "_shards": {
        "total": 42,
        "failed": 0,
        "successful": 42,
        "skipped": 0
    },
    "hits": {
        "hits": [],
        "total": {
            "value": 10000,
            "relation": "gte"
        },
        "max_score": null
    },
    "took": 134,
    "timed_out": false,
    "aggregations": {
        "ranged_documents": {
            "buckets": {
                "yesterday": {
                    "from_as_string": "2020-10-12T00:00:00.000Z",
                    "doc_count": 268300,
                    "to_as_string": "2020-10-12T12:00:00.000Z",
                    "from": 1602460800000,
                    "to": 1602504000000
                },
                "today": {
                    "from_as_string": "2020-10-13T00:00:00.000Z",
                    "doc_count": 251768,
                    "to_as_string": "2020-10-13T12:00:00.000Z",
                    "from": 1602547200000,
                    "to": 1602590400000
                }
            }
        }
    }
}

Answer 2

您可以利用derivative pipeline aggregation来实现您所期望的效果：

POST /sales/_search
{
  "size": 0, 
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "type": "toBeMonitored"
          }
        },
        {
          "term": {
            "status": true
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "now-1d/d",
              "lte": "now/h"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "histo": {
       "date_histogram": {
        "field": "@timestamp",
        "fixed_interval": "1h"
      },
      "aggs": {
        "successDiff": {
          "derivative": {
            "buckets_path": "_count"
          }
        }
      }
    }
  }
}

在每个存储桶中，您将获得前一个存储桶中的文档计数与当前存储桶中的文档计数之间的差值。