文档建议反馈控制台
首页
学习
活动
专区
圈层
工具
MCP广场
文章/答案/技术大牛
发布
社区首页 >问答首页 >ElasticSearch可视化按字段"message“中的某个关键字分组

问ElasticSearch可视化按字段"message“中的某个关键字分组
EN

Stack Overflow用户
提问于 2020-07-22 22:08:38
回答 1查看 34关注 0票数 1

我有包含字段"message“的日志，内容片段如下：

代码语言：javascript
复制
...
com.example.exception.ServiceException: Order is already completed with different status.
    --------------------------------
{
  "errorCode" : "com.example.exception.ErrorCode.CONTROVERSIAL_PAYMENT_STATUS",
  "notification_status" : "failed", // <<<< could be other values
  "order_id" : "6b30d1dd-be97-4d52-a498-4c9d3ddea84d",
  "order_status" : "accepted" // <<<<< also could be other values
}
...

我想通过"notification_status"和"order_status"之后的单词进行聚合，这样我就可以知道这两个单词的组合比其他组合出现得更多。

我能做什么?

编辑:我认为在stack_trace字段中搜索会更好。

完整的文档如下：

代码语言：javascript
复制
{
    "_index": "application-prod",
    "_type": "application-prod",
    "_id": "1hDDeHMB_M3-uozCT51K",
    "_version": 1,
    "_score": null,
    "_source": {
        "container_id": "88078d7f80436eae2fd571663c36c935eb99b03990eff25f39b92b49c017b976",
        "container_name": "/app-container",
        "source": "stdout",
        "log": "{\"timestamp\":\"2020-07-22T23:02:00.959+00:00\",\"message\":\"Failed processing notification. NotificationDto(orderId=efbb5571-cf70-46f7-9cc9-28437394c63c, reference=null, state=error, checkoutStatus=FAIL, signature=hmac-sha256:xxxxxxx, transaction=null, subReferences={})\",\"logger_name\":\"com.example.NotificationApiService\",\"thread_name\":\"elastic-1583\",\"severity\":\"ERROR\",\"stack_trace\":\"com.example.ServiceException: Order is already completed with different status.\\n\\t--------------------------------\\n{\\n  \\\"errorCode\\\" : \\\"com.example.CONTROVERSIAL_PAYMENT_STATUS\\\",\\n  \\\"notification_status\\\" : \\\"error\\\",\\n  \\\"order_id\\\" : \\\"efbb5571-cf70-46f7-9cc9-28437394c63c\\\",\\n  \\\"order_status\\\" : \\\"accepted\\\"\\n}\\n\\t--------------------------------\\n\\tat com.example.NotificationFlowSteps.updateOrderStatusAndTransactions(NotificationFlowSteps.java:38)\\n\\tat com.example.NotificationFlowSteps$$FastClassBySpringCGLIB$$e8bc6a04.invoke(<generated>)\\n\\tat org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)\\n\\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)\\n\\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\\n\\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)\\n\\tat org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:366)\\n\\tat org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)\\n\\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)\\n\\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)\\n\\tat org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)\\n\\tat com.example.NotificationFlowSteps$$EnhancerBySpringCGLIB$$3f256eb1.updateOrderStatusAndTransactions(<generated>)\\n\\tat com.example.NotificationFlowService.lambda$processNotification$0(NotificationFlowService.java:58)\\n\\t... 7 frames excluded\\n\\tat java.util.concurrent.FutureTask.run(Unknown Source)\\n\\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)\\n\\tat java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\\n\\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\\n\\tat java.lang.Thread.run(Unknown Source)\\n\",\"ASYNC_INCLUDE_CALLER_DATA\":\"false\",\"HOSTNAME\":\"88078d7f8043\",\"ASYNC_DISCARDING_THRESHOLD\":\"-1\",\"ASYNC_MAX_FLUSH_TIME\":\"1000\",\"ASYNC_QUEUE_SIZE\":\"256\",\"env\":\"prod\",\"productName\":\"my-app\",\"ASYNC_NEVER_BLOCK\":\"false\",\"host\":\"88078d7f8043\",\"message\":\"Failed processing notification. NotificationDto(orderId=efbb5571-cf70-46f7-9cc9-28437394c63c, reference=null, state=error, checkoutStatus=FAIL, signature=hmac-sha256:xxxxxxx, transaction=null, subReferences={})\",\"timestamp\":\"2020-07-22T23:02:00.959Z\",\"productName\":\"my-app\",\"env\":\"prod\",\"severity\":\"ERROR\",\"trace\":\"\",\"span\":\"\",\"thread\":\"elastic-1583\",\"class\":\"e.v.c.c.s.NotificationApiService\",\"type\":\"APPLICATION\"}",
        "timestamp": "2020-07-22T23:02:00.959Z",
        "message": "Failed processing notification. NotificationDto(orderId=efbb5571-cf70-46f7-9cc9-28437394c63c, reference=null, state=error, checkoutStatus=FAIL, signature=hmac-sha256:xxxxxxx, transaction=null, subReferences={})",
        "logger_name": "com.example.NotificationApiService",
        "thread_name": "elastic-1583",
        "severity": "ERROR",
        "stack_trace": "com.example.exception.ServiceException: Order is already completed with different status.\n\t--------------------------------\n{\n  \"errorCode\" : \"com.example.exception.ErrorCode.CONTROVERSIAL_PAYMENT_STATUS\",\n  \"notification_status\" : \"error\",\n  \"order_id\" : \"efbb5571-cf70-46f7-9cc9-28437394c63c\",\n  \"order_status\" : \"accepted\"\n}\n\t--------------------------------\n\tat com.example.NotificationFlowSteps.updateOrderStatusAndTransactions(NotificationFlowSteps.java:38)\n\tat com.example.NotificationFlowSteps$$FastClassBySpringCGLIB$$e8bc6a04.invoke(<generated>)\n\tat org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)\n\tat org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:366)\n\tat org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)\n\tat org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)\n\tat com.example.NotificationFlowSteps$$EnhancerBySpringCGLIB$$3f256eb1.updateOrderStatusAndTransactions(<generated>)\n\tat com.example.NotificationFlowService.lambda$processNotification$0(NotificationFlowService.java:58)\n\t... 7 frames excluded\n\tat java.util.concurrent.FutureTask.run(Unknown Source)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.lang.Thread.run(Unknown Source)\n",
        "ASYNC_INCLUDE_CALLER_DATA": "false",
        "HOSTNAME": "88078d7f8043",
        "ASYNC_DISCARDING_THRESHOLD": "-1",
        "ASYNC_MAX_FLUSH_TIME": "1000",
        "ASYNC_QUEUE_SIZE": "256",
        "env": "prod",
        "productName": "my-app",
        "ASYNC_NEVER_BLOCK": "false",
        "host": "88078d7f8043",
        "trace": "",
        "span": "",
        "thread": "elastic-1583",
        "class": "e.v.c.c.s.NotificationApiService",
        "type": "APPLICATION"
    },
    "fields": {
        "timestamp": [
            "2020-07-22T23:02:00.959Z"
        ]
    },
    "highlight": {
        "log": [
            "NotificationDto(orderId=efbb5571-cf70-46f7-9cc9-28437394c63c, reference=null, state=error, checkoutStatus=FAIL, signature=hmac-sha256:xxxxxxx, transaction=null, subReferences={})\",\"logger_name\":\"com.example.NotificationApiService\",\"thread_name\":\"elastic-1583\",\"severity\":\"ERROR\",\"stack_trace\":\"com.example.exception.ServiceException: @kibana-highlighted-field@Order@/kibana-highlighted-field@ @kibana-highlighted-field@is@/kibana-highlighted-field@ @kibana-highlighted-field@already@/kibana-highlighted-field@ @kibana-highlighted-field@completed@/kibana-highlighted-field@ @kibana-highlighted-field@with@/kibana-highlighted-field@ @kibana-highlighted-field@different@/kibana-highlighted-field@ @kibana-highlighted-field@status@/kibana-highlighted-field@.\\n\\t--------------------------------\\n{\\n  \\\"errorCode\\\" : \\\"com.example.exception.ErrorCode.CONTROVERSIAL_PAYMENT_STATUS\\\",\\n  \\\"notification_status\\\" : \\\"error\\\",\\n  \\\"order_id\\\" : \\\"efbb5571-cf70-46f7-9cc9-28437394c63c\\\",\\n  \\\"order_status\\\" : \\\"accepted\\\"\\n}\\n\\t--------------------------------\\n\\tat com.example.NotificationFlowSteps.updateOrderStatusAndTransactions(NotificationFlowSteps.java:38)\\n\\tat com.example.NotificationFlowSteps$$FastClassBySpringCGLIB$$e8bc6a04.invoke(<generated>)\\n\\tat org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)\\n\\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)\\n\\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\\n\\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)\\n\\tat org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:366)\\n\\tat org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)\\n\\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)\\n\\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)\\n\\tat org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)\\n\\tat com.example.NotificationFlowSteps$$EnhancerBySpringCGLIB$$3f256eb1.updateOrderStatusAndTransactions(<generated>)\\n\\tat com.example.NotificationFlowService.lambda$processNotification$0(NotificationFlowService.java:58)\\n\\t... 7 frames excluded\\n\\tat java.util.concurrent.FutureTask.run(Unknown Source)\\n\\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)\\n\\tat java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\\n\\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\\n\\tat java.lang.Thread.run(Unknown Source)\\n\",\"ASYNC_INCLUDE_CALLER_DATA\":\"false\",\"HOSTNAME\":\"88078d7f8043\",\"ASYNC_DISCARDING_THRESHOLD\":\"-1\",\"ASYNC_MAX_FLUSH_TIME\":\"1000\",\"ASYNC_QUEUE_SIZE\":\"256\",\"env\":\"prod\",\"productName\":\"my-app\",\"ASYNC_NEVER_BLOCK\":\"false\",\"host\":\"88078d7f8043\",\"message\":\"Failed processing notification. NotificationDto(orderId=efbb5571-cf70-46f7-9cc9-28437394c63c, reference=null, state=error, checkoutStatus=FAIL, signature=hmac-sha256:xxxxxxx, transaction=null, subReferences={})\",\"timestamp\":\"2020-07-22T23:02:00.959Z\",\"productName\":\"my-app\",\"env\":\"prod\",\"severity\":\"ERROR\",\"trace\":\"\",\"span\":\"\",\"thread\":\"elastic-1583\",\"class\":\"e.v.c.c.s.NotificationApiService\",\"type\":\"APPLICATION\"}"
        ],
        "stack_trace": [
            "com.example.exception.ServiceException: @kibana-highlighted-field@Order@/kibana-highlighted-field@ @kibana-highlighted-field@is@/kibana-highlighted-field@ @kibana-highlighted-field@already@/kibana-highlighted-field@ @kibana-highlighted-field@completed@/kibana-highlighted-field@ @kibana-highlighted-field@with@/kibana-highlighted-field@ @kibana-highlighted-field@different@/kibana-highlighted-field@ @kibana-highlighted-field@status@/kibana-highlighted-field@.\n\t--------------------------------\n{\n  \"errorCode\" : \"com.example.exception.ErrorCode.CONTROVERSIAL_PAYMENT_STATUS\",\n  \"notification_status\" : \"error\",\n  \"order_id\" : \"efbb5571-cf70-46f7-9cc9-28437394c63c\",\n  \"order_status\" : \"accepted\"\n}\n\t--------------------------------\n\tat com.example.NotificationFlowSteps.updateOrderStatusAndTransactions(NotificationFlowSteps.java:38)\n\tat com.example.NotificationFlowSteps$$FastClassBySpringCGLIB$$e8bc6a04.invoke(<generated>)\n\tat org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:771)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)\n\tat org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:366)\n\tat org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118)\n\tat org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)\n\tat org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:749)\n\tat org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:691)\n\tat com.example.NotificationFlowSteps$$EnhancerBySpringCGLIB$$3f256eb1.updateOrderStatusAndTransactions(<generated>)\n\tat com.example.NotificationFlowService.lambda$processNotification$0(NotificationFlowService.java:58)\n\t... 7 frames excluded\n\tat java.util.concurrent.FutureTask.run(Unknown Source)\n\tat java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.lang.Thread.run(Unknown Source)"
        ]
    },
    "sort": [
        1595458920959
    ]
}
regex
elasticsearch
elasticsearch-aggregation
EN

回答 1

Stack Overflow用户

发布于 2020-07-23 03:18:20

您可以使用下面的模板，其中我使用了示例映射、文档、我使用的聚合查询以及最后的响应：

映射：

代码语言：javascript
复制
PUT my_error_index
{
  "mappings": {
    "properties": {
      "errorCode": {
        "type": "keyword"
      },
      "notification_status": {
        "type": "keyword"
      },
      "order_id": {
        "type": "keyword"
      },
      "order_status": {
        "type": "keyword"
      }
    }
  }
}

请注意，go您也可以通过使用multi-field来使用text，但是对于查询的性质，我需要使用聚合查询，从而使用keyword类型。

示例文档：

代码语言：javascript
复制
POST my_error_index/_doc/1
{
  "errorCode" : "com.example.exception.ErrorCode.CONTROVERSIAL_PAYMENT_STATUS",
  "notification_status" : "failed",
  "order_id" : "6b30d1dd-be97-4d52-a498-4c9d3ddea84d",
  "order_status" : "accepted"
}

POST my_error_index/_doc/2
{
  "errorCode" : "com.example.exception.ErrorCode.CONTROVERSIAL_PAYMENT_STATUS",
  "notification_status" : "success",
  "order_id" : "6b30d1dd-be97-4d52-a498-4c9d3ddea85d",
  "order_status" : "rejected"
}

POST my_error_index/_doc/3
{
  "errorCode" : "com.example.exception.ErrorCode.CONTROVERSIAL_PAYMENT_STATUS",
  "notification_status" : "success",
  "order_id" : "6b30d1dd-be97-4d52-a498-4c9d3ddea86d",
  "order_status" : "accepted"
}

POST my_error_index/_doc/4
{
  "errorCode" : "com.example.exception.ErrorCode.CONTROVERSIAL_PAYMENT_STATUS",
  "notification_status" : "failed",
  "order_id" : "6b30d1dd-be97-4d52-a498-4c9d3ddea87d",
  "order_status" : "rejected"
}

注意，我用failed和success的notification_status以及accepted和rejected值的order_status构造了4个不同的文档

聚合查询：

代码语言：javascript
复制
POST my_error_index/_search
{
  "size": 0,
  "aggs": {
    "my_notification_status": {
      "terms": {
        "field": "notification_status",
        "size": 10
      },
      "aggs": {
        "my_error_status": {
          "terms": {
            "field": "order_status",
            "size": 10
          }
        }
      }
    }
  }
}

请注意，我已经将Terms Aggregation Query用于notification status，并进一步将另一个Terms Aggregation Query链接到它以用于order_status

响应：

代码语言：javascript
复制
{
  "took" : 198,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 4,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  },
  "aggregations" : {
    "my_notification_status" : {
      "doc_count_error_upper_bound" : 0,
      "sum_other_doc_count" : 0,
      "buckets" : [
        {
          "key" : "failed",                          <---- For Failed notification status
          "doc_count" : 2,
          "my_error_status" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "accepted",
                "doc_count" : 1
              },
              {
                "key" : "rejected",
                "doc_count" : 1
              }
            ]
          }
        },
        {
          "key" : "success",                         <--- For success notification status
          "doc_count" : 2,
          "my_error_status" : {
            "doc_count_error_upper_bound" : 0,
            "sum_other_doc_count" : 0,
            "buckets" : [
              {
                "key" : "accepted",
                "doc_count" : 1
              },
              {
                "key" : "rejected",
                "doc_count" : 1
              }
            ]
          }
        }
      ]
    }
  }
}

如果这有帮助，请告诉我！

票数 0
EN
页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：

https://stackoverflow.com/questions/63036207

复制
相关文章

相似问题

领券

腾讯云开发者

扫码关注腾讯云开发者

扫码关注腾讯云开发者

领取腾讯云代金券

热门产品

热门推荐

更多推荐

Copyright © 2013 - 2026 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有 

深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 粤公网安备44030502008569号

腾讯云计算（北京）有限责任公司 京ICP证150476号 |  京ICP备11018762号

问题归档专栏文章快讯文章归档关键词归档开发者手册归档开发者手册 Section 归档